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ABSTRACT 



This report discusses the goals, procedural format, 
and outcomes of a summer research training institute conducted by the 
Bureau of Educational Research and Testing Services at the University 
of New Hampshire. Trainees at the institute studied (1) the problems 
inherent in evaluating the particular educational problem with which 
the trainee was concerned, (2) major alternatives of educational 
research met hcdologies, and (3) communications techniques applicable 
to educational decision making. Each trainee also used modern data 
processing equipment for encoding and utilizing research data, and 
read current educational literature related to his own research 
projects. The report contains institute time schedules, course 
offerings, book lists, samples of various materials prepared for the 
institute, and a table depicting participants' progress. (LLR) 
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REPORT ON THE HEN SOMMER INSTITUTE - 1968 



The Bureau of Educational Research and Testing Services, a sub-unit of the Department 
of Education, University of New Hampshire , conducted a six week research training 
Institute during the period July 1st to August 9th, 1968. The grant permitted the 
training of thirty participants from any of the six New England states. 

This research training institute was funded by the Department of Health, Education 
and Welfare Research Training Division. The Institute had as its major purposes 
the five following goals: 

1. The first major theme was the study of the problems inherent in 
evaluating the particular educational problem with which the trainee 
is concerned . 

2. The second major theme was the study of the major alternatives open 
to the educator in terms of educational research methodologies; for 
instance, the experimental approach as typified by the work .of 
Campbell and Stanley, or the context, input, process and production 
model (CIPP) as presented by Daniel L. Stufflebeam. In either model 
approach, the program review and evaluation technique (PERT), presently 
being developed for education by Desmond Cook, will be advocated as the 
method of organizing. 

3. The third major theme was the study of communications techniques 
applicable to proper implementation of the decision making process 
at various levels of the educational system. 

4. The fourth major theme was the use of modem data processing equipment 
to facilitate the encoding and utilization of research data. 

5. The fifth major theme was the reading of current educational literature 
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vrith emphasis on the application o£ literature relevant to the research 
project with which the trainee is involved. 

Because of the lateness of the funding of this research institute, it was not possible 
to obtain thirty applicants. Twenty-seven were originally accepted; one withdrew 
subsequent to the first week. 

The participants from the New England states were as follows: Maine - six, Vermont - 

two. New Hampshire - fourteen, Rhode Island - none, Connecticut - one, and 
Massachusetts - three. 

The Institute's daily instructional schedule went from 8:30 in the morning until. 

4:00 P. M. and was often augmented by evening sessions and consultation meetings with 
the principal instructors in the late afternoon. Two faculty members of the 
University of New Hampshire Bureau of Educational Research and Testing Services were 
principally responsible for the operation of this institute - Dr. Gilbert R. Austin, 
Director, and Dr. Albert R. Elweil, Assistant Director. Dr. Austin is a member of 
the Department of Education and Director of the Bureau of Educational Research 
and Testing Services at the University of New Hampshire. Dr. Elweil is a member 
of the Department of Education and Director of Research and Development at the 
Bureau. Mr. James Estes, an instructor in the Department of Mathematics, conducted 
the afternoon course in computer programming. The Institute was conducted on a 
day to day basis in the following manner: 

8:30 - 10:00 A.M. Dr. Austin and Dr. Elweil jointly presented material 

relative to educational research. This involved 




instruction in statistics, experimental design and 
research methodologies. 
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10:00 - 10:30 A.M. 
10:30 - 11:45 A.M. 



12:00 - 1:00 P.M. 
1:00 - 2:30 P.M. 

2:30 - 4:00 P.M. 



Participants given a morning break 
Dr. Austin and Dr. Elwell continued the morning's 
presentation and encouraged group discussion with the 
participants with special reference to the application of 
this material to their projects. 

Lunch 

Library research, talk with major professors, or Mr. 
Estes, or work on own projects. 

Met with Mr. Estes and were concerned with how to run 
a computer and how to write computer programs. This 
included hands-on operation on both the IBM 1620 and 
the IBM 360, as well as a remote terminal system for 
the 360. 



Each institutee received fjix graduate credits for his participation In this six- 
weeks institute. The course numbers were Education 881: Methods and TCechniques of 

Educational Research . This course was a critical study of the principal methods 
employed in the investigation of educational problems and an evaluation of the 
procedures and standards used for reportings and findings; and Education 882 : 

Research Problems in Education . This course was to orient the trainees to the 
variety of educational research. In the latter portion, each trainee was expected 
to develop a research proposal which would have practical importance and relevance to 
his home school district, or to evaluate a project presently being studied. 

In all of the above, the instructional staff was involved an a full time basis. 
Because we were particularly concerned that the program should meet the needs and 
.expectations of the trainees , the program segments were evaluated bi-weekly by the 
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trainees . It became apparent as a result of these evaluations that certain changes 
needed to be made in the subject-content and instructional sequence of the Institute, 
and the methods by which they were to be implemented. A copy of these evaluation 
forms will be found in the appendix of this report. One of the basic statements in 
the original proposal for this institute was that it should be team taught. The 
staff attempted to do this, to the degree that it was possible, and to interact among 
themselves and with the enrollees. It is the opinion of the. author that this was 
more successfully done this year than in the 1967 Summer Institute. 

In addition to the questionnaire handed out each two weeks, the institutees were 
given a pre-test and a post-test based upon conceptual knowledge of statistics, 
experimental design, and research methodology held by the trainees at the beginning 
of the Institute. This pre-test - post-test consisted of a series of questions for 
which the semantic differential* technique was used to estimate the degree of 
applicable knowledge on a bi-polar scale range from 'unfamiliar' to 'competent'. 

In this instance the word 'competent' meant not only that they had acquired knowledge 
but they could use it in a working situation. The degree of knowledge of the 
institutees in the pre-test was very low; the mean for that was 36.66 with a minimum 
possibility of 22 points. The final (post-test) mean was 98. On an individual basis 
an average gain of 60.08 was computed, which is very statistically significant at 
greater than .01 level. A chart found in the appendix of this report will present 
these data for further clarification. The pre-test responses indicated that the 
institutees had some knowledge of the group of statistics which are concerned with 
measures of central tendency. They had almost no knowledge of such things as 
evaluation models, parametric or non-parametric statistics, multivariate statistics, 
computer analysis, analysis of variance or co-variance and program and evaluation 

review technique (PERT) . It was in these latter areas that the most apparent gains 

O 

by the trainees. 
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The major requirement of the six-week Institute for each participant was the creation 
of a research proposal that was educationally relevant to the problems in his school 
district, or educational agency. In the appendix of this report will be found a copy 
of the title of each of the proposals that the institutees created. It was apparent 
from reading these proposals that the trainees had made a very significant amount of 
growth, particularly in the use of program evaluation and review technique (PERT) 
and the use of the evaluation models. 

The participants apparently came to this program expecting merely to be taught 
statistics and the^ be asked to apply them to their research models. The approach 
taken by the instructional staff was to stress the development and implementation of 
a model first and then to explore the appropriate types of statistics against these 
models. We found that the institutees experienced some difficulty in adjusting to 
these expectations. To restate this another way, they came to the Institute with 
57 the preconceived notion that they would focus almost totally on statistics while the 
major professors were concerned with the ability of the institutees to first describe 
and to document what it was that they wanted to research, and to examine and specify 
alternate design strategies and, finally, to employ appropriate statistical analyses 
and facilitate their judgment of project results. 

In addition in the appendix will be found the titles of each of the institutees 
several computer programs with an indication of what kind of output they create in 
addition to this in the appendix one will find a variation of mimeographed documents 
that were passed out to the institute* trainees. 

Three major texts were used in this institute: 



1. Statistical Analysis in Psychology and Education - George A. Ferguson 



2. An Introduction to Educational Research - Robert M. W. Travers 




3. Digital Computer Programming - Thatcher and Capato 
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In addition, extensive use was made of N. E. Gage's Handbook of Research on 
Teaching » and each participant was provided a copy of the Campbell and Stanley 
chapter from that text, which is now available under separate cover. 

During the six weeks of this Institute the facilities at the Bureau of Educational 
Research and Testing Services were extensively used. A formal field trip was made 
to the Bureau where its staff made a presentation on the use of a variety of forms 
and the use of optical scanning equipment. This was an attempt to update the people 
factually and also their abilities to think about how information might be collected 
and processed • In addition to the daily instructional format the participants were 
presented 10 guest lecturers - six of whom were included in the original proposal - 
and four guest speakers - two from the New Hampshire State Department of Education, one 
from a public school system in the state of New Hampshire, and the Director of the 
Computation Center of the University of New Hampshire. 

The first lecturer of the Institute was Dr. Daniel Stufflebeam, who presented material 
on July 9th and 10th, relative to his CIPP model, (Context, Input, Process and 
Product Evaluation). This seemed to be a very meaningful presentation to the institutees* 
since it was not statistical in nature, but was oriented toward helping them ask 
appropriate questions and the planning; of strategies for seeking the answers. It was 
particularly well received because its applications to everyday school people and 
problems were easy to follow. 

Dr. Desmond Cook, on July 11th and 12th, discussed the use of PERT (Program Evaluation 
and Review Technique) as it is applied to educational research. Again, Dr. Cook was 
extremely well received. Very few of the institutees had any prior knowledge of 
PERT or its applications to educational research. Dr. Stufflebeam f s and Dr. Cook's 
material was extensively used by the institutees, both in their subsequent discussions. 
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On July 17th the Institutees came to the Bureau of Educational Research and Testing 
Services, where Robert Hart made - as a supplement to the ten scheduled lecturers — 
a presentation concerning the use of optical scanning equipment. 

On July 23rd, Dr. John Finger of Rhode Island College discussed problems of data 
analysis as a result of statewide testing. He assessed three major points in his 
presentations the first was a review of the problems in making definitive statements 
about a state school system as a result of statewide testing; second, the importance 
of stating clear program objectives prior to the start of a testing program; and third, 
the question of statistical, as opposed to educational, significance of test data. 

The next scheduled speaker was to be Dr. Victor Taber of the State Department of 
Education of New York to discuss the evaluation of Title I and Title III programs in 
the State of New York Department of Education. Becuase of personal problems. Dr. 

Taber could not attend. He did send in his stead Miss Priscilla Hayward, his 
assistant director and the coordinator of assessment of Title I. Miss Hayward 
covered very thoroughly how New York is attempting to evaluate its Title I projects. 
The institutees were particularly concerned and impressed with the state T s attempt 
to establish base line data and a cutoff point which would be used as an indicator 
of problem areas, and where Title I was, or was not, being successful in its attempts 
to improve education. 

On July 29th, Dr. Richard Neville spoke on the role of the supervisor and his 
responsibilities in the area of educational research. He focused on three major 
points that the institutees reacted to most positively. The first was the role of 
the principal in the elementary school - does he make a difference? The second 
was the role of the elementary school principal as a producer of change, and the 
role of the school as a mass production versus unit production oriented system. 
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Finally, he discussed the question of purpose and product or efficiency and 
effectiveness. The above points were made very cogently and were well received. r f . 

On July 30th, Dr. John Cawley spoke on the question of the interpretation of test data 
and from what perspective one might look at these results. Dr. Cawley's two major 
points were that there are many ways of looking at data, and it being particularly 
crucial as to which set of data you use as your reference, and second, the question 
of alternative methods of recording teacher-pupil interaction using basic materials 
and evaluation techniques. 

On August 1st, Mr. James Carr and Mr. Joseph Cannistraro spoke to the Institute. Mr. 

Carr is Consultant (Guidance, Counseling and Testing), New Hampshire State Department 
of Education. His discussion focused first on an historical perspective to the 
statewide testing program in New Hampshire. The second issue was a question of "where 
do we go from here? 11 , centering on the discussion of the broadening of the statewide 
program. The third area of concern was the role of leadership that the State 
Department of Education will play in the use of test results, particularly as these 
results are used to help school personnel make guidance, instructional and 
administrative judgments about their schools. 

Mr. Joseph Cannistraro^ Guidance Counselor at the Rundlett Junior High School in 
Concord, New Hampshire, reviewed his use of test results at Rundlett, and elaborated 
on their wide use of stanines both for grading and for other systems of reporting 
student ability or achievement. He also discussed uses of the data for grouping childreq 
by ability. 

On August 6th, Mr. Stuart Pickard (Director, Planning, Development and Evaluation), 
Division of Instruction, New Hampshire State Department of Education, made four 
— points of interest to the institutees. The first was an overview of the wide 
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variety of excellent sources of information about education research, such as the 
Educational Index and the Review of Educational Research , His information about the 
ERIC materials was particularly relevant, and the institutees should be able to make 
good use of the materials in the future. His third point was a discussion of the 
continuing concern that State Departments of Education have for the evaluation of 
Title I and Title III projects, which is very relevant at this time. His fourth major 
point was the concern expressed in terms of the reasons proposals are rejected and 
the need for stating educational objectives in behavorial terms. 

On August 7th, Mr. Richard Burrows, Acting Director of UNH Computation Center, spoke 
to the Institute on the updating of the status of the University's Computation 
Center, and particularly the use of remote terminals. Secondly, he reviewed present 
cost factors, and discussed the future of computer science and its applications to 
educational problems. 

In all of the above, with the exception of the four external speakers, the day 
presentation schedule was modified. The speakers in general made a presentation 
from 8:30 to 10:00 AM, and in the case of Stufflebeam and Cook, they again made a 
presentation in the second half of the morning. Then the Institute met from 1:00 
to 2:30 PM for a discussion and answer period with the speakers. • With the other 
four speakers the morning session was used for making a presentation, and from 
1:00 to 2:30 in the afternoon, the lecturers served as a resource person to answer 
questions that were of particular interest to the Institute. 

In summary then, it seems fair to state that the institute achieved many of its 
goals and objectives. The final judgement, however, of the effort of the institute 
will have to wait on the activities and research efforts made by the trainees them- 
selves to facilitate this. The director and associate director of the institute are 
planning a vigorous follow up effort to continue the relationships developed during 
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As noted earlier in this report in the appendix will be found a table 
titled Statistical Data and Tests for Significant Gain in Perceived 
Knowledge . It would seem to the writer that this table deserved care- 
ful studying for it seams to substantiate markedly the opinion that the 
trainees did in fact aquire a considerable amount of valuable knowledge. 
Of the twenty- two areas of knowledge measured only five did not show a 
statistically significant gain over the six weeks at the institute. It 
is further encouraging that those areas which are most statistically 
significant fall into the areas most stressed in the original five 
objectives of the institute. 

In addition, in the appendix of this report will be found a sample of a 
variety of types of materials that were prepared for this institute. 
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Richard Cardner: Evaluation of proposal ,e A PROGRAM FOR EVALUATION OF A 

SCHOOL READINESS PROGRAM" 



A) Strengths 

1. Title reasonably good 

2. General introduction to the problem - very good 

3. Reference to literature - very good 

4. Criteria for evaluation - very good 

5. Use of 'PERT* is very impressive 

6. Awareness of the difficulty of the research - excellent 



Fernand Prevost: Evaluation of proposal "AN EXPERIMENTAL APPROACH TO 

TO THE TEACHING OF PRIMARY SCHOOL MATHEMATIC S" 



A) Overall Evaluation 

Excellent, well documented and carefully thought out 

B) Strengths 

!♦ Good title 

2 . Excellent introduction 

3. Design seems to be well thought out 

4. Very interesting and clearly and candidly stated 

5. Results or expected results based on James Lucas 1 s study 

6. Use of "PERT" excellent 

7. Proposed use of statistics appropriate 



David Mann: Evaluation of Proposal "EVALUATION OF PRESENT SYSTEM OF 

STUDENT SCHEDULING AT EXETER HIGH SCHOOL IN ORDER TO 
RECOMMEND DESIRABLE CHANGES" 



A) Overall Evaluation 

A well thought out project 

B) Weaknesses 

1* No reference to what others have learned in this same area. 
Should propose some study of the literature, 

C) Strengths 

1 , Statement of areas of examination - very good 

2, Title quite clear 

3, Time schedule seems quite realistic 



EVALUATION RESEARCH 



Hugh Holt: Evaluation of proposal "EXPlkMENTAL 

PROPOSAL AND DESIGN" 



A) Overall Evaluation: 

An excellent proposal, but I still don't know where and on what population 
this is going to be tried. 

B) Weaknesses 

1. Title is unclear. I don't know what the study is about. 

2. The second section called proposal outline and terms does not give 

a setting of the study. 

C) Strengths 

1. The first section called problem context is exceptionally well done. 

2. The three statements of premises are particularly excellent 

3. The section on selection is quite good 

4. Hypothesis are well stated and clear 



Donald Ellis: Evaluation of proposal "NON-GRADING MY ELEMENTARY DISTRICT" 



A) Overall Evaluation 

The objectives are stated adequately but no indications are given as to 
how they will be achieved, what evaluation proceedures will be used, 
and there are no limitations stated for the study. 

B) Weaknesses 

1. I don't know xvhat "my elcrientary district" means 

2. Statement of the problem very general - no definition of the term 

non-graded 

3. I do not understand the inclusion of the statement of the high 
school principal in a report that is concerned with non-grading 
of elementary school primary grades. 

C) Strengths 

Under the heading "setting" the location of the school is quite adequately 
stated. 

D) Again all things cannot be done in one study for all children 



Ray Brennick: Evaluation of proposal "OPERATION BOOTSTRAP" 

A) Overall Evaluation: 

Quite a good project 

Introduction good, statement of problem is reasonably clear 

B) Weaknesses : 

1. Does not define terms such as 
Comprehensive, exceptional students, etc*... 

2. Deals in too general a way in how he is going to evaluate the study 

3. Sets up few limitations for himself and his proceedures are too 
general 

4. There Is no actual information given on how the project will be 
evaluated 

5. Does not cite who will be responsible for building these evaluative 
instruments 

6. Does not cite any literature which is related to this study 

C) Recommendations : 

That Mr. Brennick zero in on just one of the many areas he has indicated 
here and work on it much more intensively. This I think is best related 
to his concern for all of the students and yet we all know that nobody 
can do everything for all students* 



R. C. Bailey; Evaluation of proposal "A COMPARISON OF CONVEYED AND NON — 
CONVEYED STUDENTS AND THEIR SCHOLASTIC ACHIEVEMENT" 



A) Overall Evaluation 
Rather a good proposal 

B) Weaknesses 

1, The limitations of the study are a bit fuzzy 

C) Strengths 

1. Terms seem to be adequately defined 

2. Reasonably adequate introduction 

3. Statement of the problem is reasonably clear 

4. Justification of the study is good 

5. Setting of the study Is clearly defined 

6. The proceedures methods that he is going to employ to proceed with 
the study seem quite clearly stated. 

D) Recommendations 

1. The word conveyed probably ought to be changed to transported or 
bus transported 



John C. Emerson; Evaluation of proposal "A PILOT STUDY: THE EFFECTS OF 

PROGRAM SEQUENCE ON SUCCESS IN SECONDARY SCHOOL MATHEMATICS" 



A) Overall Evaluation 

In general - very good. The proposal is basically sound 

B) Strengths 

1« Title - interesting and well stated 

2. Statement of problem - quite adequate 

3. Reference to literature on the subject - good 

C) Recommendations 

It seems to be rather sketchily presented, and needs a great deal more 
detail, than is presently offered. 



Donald MacLean; Evaluation of proposal "DESIGN FOR THE EVALUATION OF A 
PUBLIC SCHOOL" 



A) Overall Evaluation 

Report is not specific enough. I know the general areas and categories 
that work will be done in but no information is given to me on the 
limitations of the study, defining the proceedures, no review of the 
literature or reference to literature is made. 

B) Weaknesses 

1. How and by what means the research will proceed, is not clearly 
stated. 

2. Title - very broad - does not state what is to be evaluated 

3. Area of standards are broad categories, need to be more specifically 
stated 

C) Strengths 

1. Use of "PERT" technique - very interesting 



George Puts: Evaluation of Proposal "A SCENARIO FOR HIGHER EDUCATION 11 



A) Overall Evaluation 
Excellent 

Ability to implement it or evaluate it, unknown 

Realistic rating - let's take a small part of it, and try working it out 

B) Strengths 

1. General introduction - very good 

2. Statement of the problem - fascinating 

3. A very provocative paper; its implications are far reaching 

4. Title - intriguing 



Stanley Hall: Evaluation of proposal "AN INTERMEDIATE CRITICAL THINKING 

EVALUATION* 1 



A) Overall Evaluation 

Very good, a very comprehensive job 

B) Weaknesses 

1* Title not clear. I think he means the development of an instrument 
to test critical thinking. 

C) Strengths 

1 . Statement of the problem - quite good 

2. Familiarity with the research and related literature - excellent 

3. Proceedures - quite adequate 



David Beisel: Evaluation of proposal "not given" 



A) Overall Evaluation 

Needs careful work on the limitations of the study, and proceedures to 
be used 



B) Weaknesses 

1. Proposal is not adequately stated, although interesting 

2. Two questions asked at the end of the report are very pertinent 
but very difficult to assess. 

C) Strengths 

1. Introduction adequate - his own philosophy. Clarification of his 
own model for cirriculum development through group dynamics . 

2. Use of "PERT" work breakdown structures - interesting. 

:RJLC 



Leland Churchill: Evaluation of proposal "GRADE TEN READING PROJECT" 



A) Overall Evaluation 
Good 

B) Weaknesses 

1* Title too general 

First question posed is not clear, there is no obvious comparison 
with another group being made. It is not clear just what treatment 
will be given to Group 2 and Group 3 that may be compared with the 
treatment given to Group 1. 

C) Strengths 

1. Statement of the problem, quite general, but reasonably clear. 

2. Use of the factorial design very interesting if it is appropriate 



Anne K. Lee: Evaluation of proposal "DEVELOPMENT OF A MATHEMATICS PLACEMENT 

TEST FOR THE WOODSTOCK COUNTRY SCHOOL" 



A) Weaknesses 

1. There does not seem to be any amount of limitation 

2. The procaedures for actually persuing the study are not too 
clearly spelled out 

B) Strengths 

1. Title is appropriate 

2. Introduction is adequate 

3. Statement of the problem is reasonably clear 

4. Sequence of the operation proposed seems to be adequate 

C) Recommendations 

The proposal seems rather broad and perhaps should be broken down into 
smaller pieces, to be researched, each individually 



Charles St. Paul and John Kelleher: "WORK-STUDY PLAN, LAWRENCE HIGH SCHOOL" 



A) Overall Evaluation 

It is not very specific 

B) Weaknesses 




1. Title - quite vague 

The term work-study program is not particularly well defined for 
this local Lawrence High School situation 



continued Charles St. Paul and John Kelleher 



2. The specific goal of this project are not well stated 

3. The methods of assessment are not really stated at all 

4. There are few limitations stated and the proceedures for 
evaluation are inadequately stated 

C) Strengths 

1. The introduction to the problem is reasonably straightforward 

2. The reference to related literature is adequate 

3. Attempt at task diagram - very good 

D) Recommendations 

Define goals more clearly ie., is it good only for potential drop-outs 
to have work-study programs or is it good for others 



William C. Lary; Evaluation of proposal "TO SYSTEMATICALLY REGENERATE 
A HUMANISTIC SCHOOL PHILOSOPHY AT HOLLIS HIGH 1 ' 



A) Overall Evaluation 

The general format of the proposal is quite good, and shows a great 

deal of time and thought and energy 

3) Strengths 

1. Breaking it down Into phase 1 and phase 11 seems reasonable 

2. General introduction is quite adequate 

3. The attempt at using a work breakdown structure is excellent 

C) Recommendations 

1. The purpose which states 11 to study human nature in the context of 
a public school" Is a very broad, even though laudable idea; it 
needs to be narrowed and tightly defined 

2. The development of an instrument to say what is the present 
philosophy and how perhaps it should be changed to become more 
humanistic will be very difficult 
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Roger L. Marchand: Evaluation of proposal "PILOT STUDY OF THE EFFECTS OF 

REMEDIAL READING AND COUNSELING IN COMPARISON TO JUST 
REMEDIAL READING IN RELATIONSHIP TO THE IMPROVEMENT OF 
READING SKILLS AND COMPREHENSION OF CHILDREN, TWO OR 
MORE YEARS BELOW CAPACITY LEVEL IN READING" 



A) Overall Evaluation 
Very good 

B) Strengths 

1. Reading disability well defined 

2. Cood clarification of problems associated with reading difficulties 

3. Good review of the literature on this subject 

4. Proceedures and problems well documented 

C) Recommendations 

1, Title ought to be abbreviated somewhat 



Alvord Graham: Evaluation of proposal "TEACHER ASSIGNMENT FOR PRE-DROPOUTS" 

A) Overall Evaluation 

Title not clear, don't know what the study is about as a result of 

reading it* 

B) Weaknesses 

1. Statement of problem is a bit hazy 

2. Definitions of teachers are more encouraging to pre-dropouts than 
others, and the word staying power of teachers 

3. It is not really clear as to how the teacher variable here is going 
to be assessed 

4. The proposal is very general 

5. There is no justification of the study, no setting of the study, 
no limitations 

6. Troceedures are very sketchy 

7 « No indication of any review of literature 



Eleanor Terreson: Evaluation of proposal "EDUCATIONAL INNOVATIONS SUMMER 1968" 

A) Overall Evaluation 

It is difficult to understand what is unique about this project, as 
opposed to what in general most teachers would hope to be doing* 
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continued Eleanor Terreson 



B) Weaknesses 

1. It is not clear why one finds the series of dittoed sheets in. the 
front of the proposal 

2. Unclear about what Miss Terreson means by "I am a believer in the 
whole child” 

3. No stated introduction 

4. There is no statement of a prob3.em other than a very broad and 
very general concern for a great variety of things . 

5. Slight hint of the setting of the study 

6. There are no limitations 

7 . There are no set of proceedures by which any evaluation of what’s 
going on is going to be made 

No reference to a review of any kind 



Bruce J. Kinney: Evaluation of proposal "AN APPROACH TO INSTRUCTION AT 

THE KINDERGARTEN LEVEL” 



A) Overall Evaluations 

Basic ideas hehind the proposal are sound but there is a great deal of 

detail lacking - really quite general 

B) Weaknesses 

1. I question the statement cutting down the number of slow learners” 
I don’t think any study cuts down the number of slow learners. It 
may identify them more adequately and thereby make it possible to 
prescribe aid for them 

2. There are no limitations stated 

3. The person or persons responsible for this project are in no way 
indicated nor is their position in the line staff indicated 



Gordon Flight: Evaluation of proposal ”AN ORGANIZATIONAL PLAN TO PROMOTE 

INDIVIDUALIZED INSTRUCTION IN GRADES 4 THROUGH 6 IN SOCIAL 
STUDIES IN THE GROVETON, N. H. SCHOOLS” 



A) Overall Evlauation 
Excellent 

B) Strengths 

1. Title reasonably good 

2. Origin of Problem - good 

3. Clarification of proposed research project > page 1, very good 

4. Impressive documentation in the beginning of the proposal 

5. Introduction - very good 



Richard E. LeCIaire: Evaluation of proposal "TESTING PROGRAM" 



A) Weaknesses 

1. Title too broad* It doesn't give any real information about the 
subject of the proposal 

2. Reason for the inclusion of the letter to Mr. Francis Wilson about 
Alden Lovell's HBW (Harcourt, Brace & World) proposal is not really 
clear to me. 

3. There are no proceedures specifically alluded to 

B) Strengths 

1. The statement of the problem is reasonably good, except that it is 
extremely broad and not narrowed down to a really researchable 
problem. 

2. There is a series of objectives which are well stated, but no real 
method of attainment of those objectives is offered. 

C) Recommendations 

Perhaps the proposal should have involved just the question that is 
offered last and that Is "can the administration, faculty and 
guidance department of Sanborn Regional High develop a testing program" 
and then pose questions about how one would persue this. 



Lionel DeLacey: Evaluation of proposal "EVALUATION OF A PRIMARY NON-GRADED SCHOOL" 



A) Overall Evaluation 

In general, pretty good 

B) Weaknesses 

1 . Title - very broad and very general 

2. The terms are not clarified nor defined ie., non-graded, student 
success suggestions 

3. Evaluation in terms of statistics where comparisons are being 
made between groups is not very clear (this is called the second 
phase of the evaluation) 

4. Few limiting factors in the study, limitations need to be more 
clearly spelled out 

5. Proceedures are somewhat vague 

C) Strengths 

1. Good introduction to the problem 

2. Three evaluation questions posed quite good 

3. Evaluation in terms of using PERT and CIPP models appropriate 



0 




Sister Jacqueline Eefcerc; Evaluation of proposal "A RESEARCH PAPER ON THE 

5 YEAR PROJECTION OF CATHOLIC SCHOOL SYSTEMS IN 
THE DIOCESE OF MANCHESTER, N. H." 



A) Overall Evaluation 
Very good 

B) Weaknesses 

1. The paper seems to raise more questions than answers., It does not 
pose a method of attack . These need to be stated. 

C) Strengths 

1. Good title 

2. Statement of the problem - quite direct and adequate. The statement 
"we must identify our aims in Catholic Education" it seems to me 

is one of the most crucial statements in this whole paper. 

3. Statistics and drawings at the end of the paper are interesting and 
quite factual. 

D) Recommendations 

A set of research proceedures is what probably should be the first step 
in attempting to move on this problem. 
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o{. l=?oot, 2=f ait, 3=Good, 4=Excetlent, ot UA=Not Applicable. Additionally , list in 
nattative fiotm the petceived strengths and weaknesses each ptesentation and yout sug- 
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Gil Austin — Instructional and 

Consultation 

At Elwell — n 
Jim Estes — Computer Science 6 
Pnognamming 

Richand Neville -- Univ. ofi Manyland 



John Cawley — Univ. ofi Connecticut 



Jim Coma — N.H. State Vept. Education 



Joe CamistnaAo — Rundlett JHS 



Stu VickoAd — N.H. State Vept. Educ. 



TOTAL INSTITUTE EXPERIENCE 



Comments on youn. Institute Expedience 



O 

ERIC 



TABLE — Statistical Data and Tests for Significant 
Gain in Perceived Knowledge - Summer Institute 1968 





Content Area 


X-Pre 


X-Post 


D 


S d 


fc D 


P 


a. 


Evaluation Models 


1.58 


4.83 


3.25 


1.26 


+2.58 


>.01 


b. 


Program Evaluation and Review 


1.71 


5.04 


3.33 


1.46 


+2.28 


>.025 


c. 


Parametric Statistics 


2.00 


4.17 


2.38 


1.44 


+1.65 


n. s. 


d. 


Nonpar ametric Statistics 


1.50 


4.17 


2.67 


1.27 


+2.10 


>.025 


e. 


Multivariate Analysis 


1.17 


3.50 


2.33 


1.40 


+1 .66 


n. s. 


f . 


Measures of Central Tendency 


2.42 


5.38 


2.96 


1.73 


+1.71 


>.05 


g. 


Measures of Variation & Symmetry 


1.83 


4.54 


2.71 


1.57 


+1.73 


>.05 


h. 


Probability Theory 


2.25 


4.50 


2.21 


1.89 


+1.17 


n. s. 


i. 


Sampling & Sampling Distributions 


2.42 


4.92 


2.50 


1.47 


+1.70 


n. s. 


j. 


Hypothesis Testing 


1.88 


4.67 


2.79 


1.56 


+1.79 


>.05 


k. 


Linear Relationships 


1.38 


4.08 


2.71 


1.17 


+2.32 


>.025 


1. 


Correlation of Quantitative Data 


1.67 


4.79 


3.12 


1.15 


+2.71 


>.01 


m. 


Correlation of Qualititative Data 


1.58 


4.62 


3.04 


1.33 


+2.29 


>.025 


n. 


Internal and External Validity 


1.46 


5.04 


3.58 


1.56 


+2.29 


>.025 


o. 


Comparative Experiments 


1.50 


4.38 


2.88 


1.48 


+1.95 


>.05 


P. 


Analysis of Variance 


1.38 


4.21 


2.96 


1.27 


+2.33 


>.025 


q- 


Analysis of Covariance 


1.25 


3.88 


2.62 


1.41 


+1.86 


>.05 


r , 


Optical Scanning Document Design 


1.12 


3.00 


1.88 


1.42 


+1.32 


n. s . 


s. 


Data Processikng Configurations 


1.12 


3.96 


2.83 


1.40 


+2.02 


>.05 


t. 


Computer Programming 


1.50 


4.58 


3.21 


1.14 


+2.81 


>.005 


u . 


Instrument Design and Evaluation 


1.25 


4.46 


3.46 


1.09 


+3.17 


>.005 


V. 


Research Proposal Requirements 


1.83 


5.12 


3.29 


1.33 


+2.47 


>.025 




TOTAL INSTITUTE CONTENT 
(average scale score) 


36.67 

(1.67) 


98.00 

(4.45) 


60.08 

(2.73) 


25.76 


+2.33 


>.025 



N=24 (complete pre-post evaluation data); df=23; t- required at selected levels of 
-•‘Q-'ficance (all one-tail): .05=1.71, .025=2. 07,. 01=2. 50, and .005=2.81) 




HEW PROGRAM TITIES 



Terre son, Eleanor 



Marchand , Roger 

Churchill, Leland 

Holt, Hugh 
Lary, Bill 

St. Paul and Kelleher 
Cardner, Richard 

Beisel, David 
Bailey, Ronald 
Emerson, John 

Graham, Geoffrey 
iee, Anne K. 



O 




Grade Scores, Average (Mean) 

1. List of scores, each list labeled 

2. Mean of each list (labeled) 

3. Standard deviation of each list (Labeled) 

4. Correlation Coefficient 

Lab Assignment: Pencils, 3$ each over 5, 5£ each up to 5 
and 5; for any number of pencils 

Lab Assignment: Exam Grades - how many failed, how many 
passed, how many scored 95 or above 

Readability Formula for books for Grades 1-8, giving 
grade and interest level. 

Program showing how many students passed, how many failed 
and how many received 95 or above. 



Program giving the means, standard deviations and correlations 
of two sets of scores 

Lab Assignment - list labeled scores, compute mean, standard 
deviation, correlation coefficient. Label each. 

Final Program - Compute mean, median, mode, X and standard 
scores . 

X, Y variables w/lists, means, standard deviations, correla- 
tion coefficients. 

Program - sort cards (grade), fail/pass/pass high 

Lab: Raw score, standard score, mean, median, mode 

Final Project: Mean, standard deviation, Correlation 

Sorting Program 

Final Program: Program to calculate the mean, standard dev- 

iation, skewness, kurtosis, and standard scores, (revised) 

Program for 2 sets of scores ++ lists of scores, means and 
standard deviations, correlation coefficients 

Determining number of grade scores below 70 and above 90 

Pencil Problem 

Final Project: Mean, median, mode, standard deviation, 

standard scores, skewness and kurtosis 

Lab Assignemnt: Mean, standard deviation, correlation coeff. 

Program to compare mean,, standard 'deviation, correlation' 

coefficient. 



Page 2. 

Lee, Anne K. (con't) Program: mean standard deviation, sltewnese, kurtosis, stand- 

ard score 



Hebert, Sister Jacqueline Program for obtaining mean, standard deviation, correlation 

coefficients for 2 grades lists 

Project for obtaining scores, means, standard deviation, mode, 
median 



Hall, Stanley 



Prevost, Fernand 



Kinney, Bruce 



Alvord, Graham 



Putz, George 



Sort (ascendency) 

Scores, means, standard deviation, + correlation, standard 
scores, kurtosis and skewness 

Count 2 sets of data (could be averaged) and the highest 
score tabulated 

Given 2 sets of scores, program computes mean, standard 
deviation, standard score and correlation coefficients. 
Printout includes raw scores and standard scores. 

Mean, standard deviation, kurtosis, skewness, standard scores 

2 sets of data, mean, standard deviation, correlation coeff. 

Final Project: Kurtosis, standard scores, mean, standard 

deviation, skewness 

For two groups of scores compute the 

1. means 

2. medians 

3. modes 

4. standard deviation 

Indicate negative or positive skewness, also shape of curve 
print out raw scores and standard scores. 

Multiple means, multiple standard deviations and multiple 
correlations 

Computes a ranking for grades 



LeClaire, Richard Ranking of students by range of scores 

Multiple means and standard deviations + correlation 
coefficients 

DeLacey, Lionel Multiple means and standard deviations 

Computes means, standard deviations, medians, and modes 




H.E.W. PROJECT TITLES 



John Emerson: 

The effect of course sequence on student success in mathematics. The 
study will be aimed at determining if the placement of mathematics 
content courses at different places in the high school sequence has a 
bearing on the success of students studying mathematics. 



Gordon Flight: 

Project Abstract: 

An Organizational Plan to promote individualized instruction of social 
studies in grades IV, V, and VI in the Groveton, N. H. School. 

( The purpose is to develop an appropriate plan for organization and 
initiation ) 



Donald Ellis: 

Abstract: 

I am in the process of non-grading an elementary school district as a 
result of merging six small schools into a body of approximately 550 
students . 

Maybe we should start by just non-grading the primary? Maybe we can't 
non-grade at all. We have over a year to research and we are certainly 
going to tTy! 



Geoffrey Graham: 

Proposal : 

To study primary grade children with learning difficulties, especially 
in the motor-perceptial areas . 

A team of qualified people in medicine, speech, psychology, psychiatry, 
and physical education will help us assess pupils' individual problems. 
Instruments to measure problems and progress are being developed. 

The goal is to move from separating these youngsters from the normal 
classroom to better meeting their needs within the classroom and total 
school environment. 
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Dick LeClair: 

Testing program- Evaluation of curriculum, methods of instruction- 
modification of the instructional system. 

Marking system - simplification of determining class rank with weighted 
grades . 



Leland Churchill: 

A project which allows two types of educational treatments of reading and 
basic English to be tested for its educational and statistical significance 
using a pre-test - post- test control group experimental design. 



Sister Jaqueline Hebert: 

My project was to give as perfectly a s possible a projection of the Catholic 
education problem facing the Manchester Diocese of Education. In the next 
five years there will be inevitably a decrease in pupil enrollment, a rise 
in education cost triple if not more what is known today and the danger of 
of closing many small schools, elementary or higher. This latter step will 
definitely affect your pocketbooks. We want to prevent a major catastrophy 
affecting you . 



Hugh Holt: 

Out of motivation to really enable teachers to individualize instuction 
the proposal encompassed a grouping based upon reading ability and 
uniquely structured and for a small (2 classes at each grade level) school. 
That is, the tcp and bottom 1/4 were grouped with one teacher for cert- 
ain instruction and the middle 1/2 were the others. Homogeneous grouping 

was provided in other instructional areas etc Testing was proposed 

for achievement growth, social/emotional effects and actual individual- 
ization resulting from the grouping. 
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Charles St. Paul and John Kelleher: 

WORK STUDY PLAN 
OBJECTIVES: 

1) To decrease the drop-out rate and dicipline problems in Lawrence 
High School 

2) To attempt to make education meaningful to this type of student. 

3) Ts plac> these students in job areas which will provide them with 
the opp .rtunity for permanent employment. 

PLAN: 

The students will spend ;he A.M. session, 8 to 12, in school and 
be released for wcrk at 12:00. 



Fernand J. Prevost: 

An Experimental Approach to The Teaching of Primary School Mathematics. 



Donald MacLane: 

I ‘'PERT" ed a general evaluation of what we in my school will be involved 
with this year. I also took an area and made a "PERT" design for that 
area giving target dates and showing the way we planned to go. 



David Mann: 

Evaluation of computer scheduling of a high school - what should we be 
looking for in terms of parameters. 



Grahan Alvord: 

Pre -dropout ; will be identified (out of step with classmates), their 
experience with teachers rewarded, teachers will be identified who 
encourage the pre-dropout to remain in school, with the purpose of in- 
spiring future assignments of pre-dropouts. 
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Lional De Lacey: 

Evaluation of a Non-graded Elementary School, K-6 

The three major areas to be evaluated: 

1) Curriculum 

2) Student progress and reaction 

3) Teacher influences and reaction 

Tools - PERT, CIPP - test and survey tools 



Raymond Brennick: 

Operation Bootstrap 

A Curriculum development project - grades 7-12 including a regional 
Vocational Center as part of the comprehensive program. 



George Putz: 

A Scenario for Higher Education; A Prolegomena to a Proposal for reform. 



Anne Lee: 

A proposal dealing with plans for devising a comprehensive test in numerous 
sections, to be used in placing students from a wide variety of backgrounds 
in secondary lavel math courses at the Woodstock Country School. 

(I will not be at the I'ayland address most of the coming year, but at 
Woodstock Country School, South Woodstock, Vermont. I will however, be 
reachable indirectly and during vacations directly at Wayland.) 



David Beisel: 

Can School Communications Be Improved So That Teachers Can/Will Develop 
Positive Affects Within That School? 



AFFECTS 

School Philosophy 
Public Relations 
Team Work 

a. team planning § teaching 

b. integrate communication 

c. teacher fellow program 
Evaluation 

a. within department 

b. resident observer 



VEHICLE S 

By dept, head and teachers 
By specific persons and/or 
teacher-parent home talks 
Human relations specialists 
Sensitivity training 
Intergrated course 



( see pg. 10 of proposal ) 



1 ) 

2 ) 

3) 



4) 
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David Eeisel (cont,): 

c. pre/post student and teacher questionaire 

d. attendance 

e. cafeteria use 

f. open lab use 

g. failures and dicipline 



Stanley Hall: 

I have going a three phase project which creates, pilot studies, and 
more rigorously evaluates an •' INTERMEDIATE CRITICAL THINKING APPRAISAL ' 
for grades 7 and 8 of public schools in New Hampshire. It is like the 
Watson-Glaser Critical Thinking Appraisal in Base, theory, and form, but 
more elementary. 



Richard Cardner: 

We have established a readiness room ( Ilg § Ames - School Readiness ) 
which consists of children we feel unready (maturationally) for grade 
one activities. They are selected by a combination of: 

1) Metropolitan School Readiness Examination 

2) Gesell Developmental Examination 

3) Parental and Staff observation 

We hope to evaluate their progress ( compared to a matched group ) in 
terms of : 

1) dicipline 3) achievement 5) desire to learn 

2) adjustment 4) self concept 6) school readiness 



Ronald Bailey: 

A Comparison of Conveyed and Non-Conveyed Students and Their Scholastic 
achievement. 

This proposal is a pilot study of a transportation problem in a large 
school district in Maine. The problem is time being spent on the bus 
and does this extended time affect the learning of the students involved 
as compared to walking students. 



Eleanor Terreson: 

The project of Title I in reading in Terryville, Conn, is what I will 
be working on, in effect, all year. I am solely responsible for the 
execution of the job of helping Title I students to improve in reading 
ability, and also become better adjusted in the social and physical aspects. 

O 



/ 
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Roger Marchand: 

Pilot study comparing reading differences between children who have 
remedial reading and counseling and those who have just remedial reading. 
The purpose is to see if those who have both services show significant 
differences from those that only have the reading. 



Bruce Kinney: 

Proposal : 

•o institute a new program of instuction for the pre-first grade level 

The present program was a downward extension of grade 1 - both teachers 
and materials in to kindergarten. This needs more rethinking and piloy 
programs to change it into a program for five year olds. 



Bill Lary: 



Problem: 

To regenerate the present school philosophy at Hollis High. 

Two phases: 

1) Build instuments to evaluate staff and student opinion about human- 
ities program - administer them. 

2) Process evaluation to increase; communication 

cooperation 

effectiveness 

efficiency of the school. 




Here is the evaluation of your summer proposal. Both Dr. Elwell and I have 
read them and these are general comments. 

By way of review the things we looked for in the proposals essentially were 
the following: 

I. STATEMENT AND DEFINITION OF THE PROBLEM 

Introduction 

Statement of the Problem 

Justification of the Study 

Setting of the Study 

Limitations of the Study 

Procedures 

Definition of Terms 

II. REVIEW OF RELATED LITERATURE 

III. RESEARCH PROCEDURES 

Research Design 
Orientation of Teachers 
Experimental Procedures 

Factors Jeopardizing Internal and External Validity 

IV. INTERPRETATION OF RESULTS 

V. SUMMARY AND CONCLUSIONS 



O 




H.E.W. Institute f 68 



Problem Illustrations of Sources of Internal Invalidity - B 



Illustration Is 



An evaluator had to make a choice between SMSG and UICSM mathematics curricula for all 
ninth grades in the state of New Hampshire. An experiment was run in which the achieve- 
ment of classes studying under both curricula was compared. A sample of 200 ninth-grade 
teachers was drawn at random from the population of ninth-grade teachers in the state. 
One-hundred volunteers were taken to teach SMSG mathematics; the remaining 100 teachers 
were assigned to teach the UICSM course. The 200 experimental teachers were trained 
during the summer on their new curricula and then spent two semesters instructing their 
students. At the end of the second semester, the achievement of the 100 classes taught 
by SMSG teachers was superior in almost every respect to that of classes taught by UICSM 
teachers . 

What is probably the greatest source(s) of internal invalidity in this experiment? 



Illustration 2: 



The value of supplemental calisthenics to the physical education curriculum is being 
evaluated. Of the 500 students enrolled in an elective physical education course in a 
large high school, 50 were randomly sampled to participate in an experiment as a Single 
class. A randomly chosen 25 of these 50 students were held for 10 minutes after each 
class and put through a strenuous regimen of calisthenics. The other half of the class 
was excused. At the end of the semester the two groups (supplemental calisthenics versus 
no calisthenics) were compared on several measures of strength, agility, balance, etc. 

On one measure of Strength the average score of the 23 students in the no-calisthenics 
group who completed the course was 104.79, and the average score of the 18 students in 
tho supplemental calisthenics group who completed the course. was 128.54. Statistical 
analysis showed the supplemental calisthenics group to be unquestionably superior. 

What. is possibly the most important source of internal invalidity in this experiment? 



Illustration 3 : 

An experiment was performed to evaluate the effects of glutamic acid on the intelligence 1 
of mentally defective children. A sample of 100 congenital mental defectives was randomly 
split into an experimental and control group of 50 subjects each. Pretesting with the 
Stanford-Binet was performed on all 100 subjects. For six months the experimental subjects 
received doses of glutamic acid each day. The medication tended to make the experimental 
subjects hyperactive, distractible, and uncooperative. At the end of the experimental 
period, all subjects were tested again with the Stanford-Binet. The same researcher gave 
the pre-test, administered the medication, and gave the posttest. The following data 
were obtained: 

Experimental Group Control Group + 

Pretest Average IQ = 74.3 Average IQ = 73.9 

Posttest Average IQ = 82.6 Average IQ = 75.1 

Statistical analysis revealed that the experimental subjects made significant gains in IQ 
scores and the control subjects did not. What is probably the most important source of 
invalidity in this experiment, if such internal invalidity does exist? 




HEW - Illustrations Internal Invalidity -2- 



Illustration 4 (a real case); 



An evaluation of the effectiveness of a public versus parochial school education was 
undertaken. From a population of 60,000 Catholic school eighth-graders, 1,000 pupils 
were seledted^at ’randbm. A sample of 1,00.0 from a public school population of 80,000 
was chosen in such a way that they were matched with the Catholic pupils exactly on a 
test of non-verbal reasoning. The two matched samples were then measured on arithmetic 
achievement. . 

The average grade-equivalent score in arighmetic achievement for the 1,000 Catholic pupils 
was 10.02; the comparable average for the public-school pupils was 9.04, This difference 
was highly statistically significant , indicating that fhe parochial schools did a better 
job of instruction in arithmetic that the public schools. 

Is there a possible source of invalidity in this experiment? 



Illustration 5 ; (also a real case) ; 

A study was undertaken of the pattern of attitude change among student teachers. The 
attitudes toward teaching of 50 student teachers and the supervisory teachers were 
measured with the Minnesota Teacher Attitude Inventory before and after a semester of 
student teaching. After considerable rummaging around in the data, the researcher 
discovered that student teachers whose attitudes were initially poorer than those of their 
supervising teacher had positive attitude gains from the beginning to the end of the 
semester. The group of student teachers as a whole did not show an improvement in 
attitude . The researcher concluded that , if a student teacher is under the influence 
of a supervisor who holds a batter attitude toward education, the student teacher’s 
attitude will improve. 

Can you identify an influence in this experiment which might make this conclusion invalid? 




h 
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Correlation 

Name: Mastery Test :Qualitative 

Data (A) 



Problem 1: As most of you struggled through another section of Campbell 

and Stanley during the twilight hours, your instructional staff 
was hard at work collecting data at the outdoor swimming pool 
for input into the next mastery test. The sample data which we 
humbly submit consists of the observation of 100 Summer School 
coeds among whom were 7 very well controlled blonds, 32 

were blonds with average self-control, and 14 were (at this 
point of the summer session) very impulsive blonds. We also 
casually observed 47 brunettes in our sample of which 9 were 
very self-controlled, 26 were classified as average self-control 
and 12 were checked out to be very impulsive. Since our only 
purpose was to save you the time of screening a sample for 
your statistical problem (we don*t have time to compute it 
for you), decide on the basis of this hard-core information 
given above whether there exists a correlation between our 
sample 1 s hair color and their level of self-control as observed 
by your hard-working instructional staff (use a significance 
level of .01). 



Problem 2 



Decide on the basis of the information given in the following 
table whether there exists a correlation between a student T s 
ability and interest in extra-curricular activities at School A- 
.05 level of significance). 





Low 


Ability 

Average 


High 


Low 


28 


17 


15 


Average 


20 


40 


20 


High 


12 


28 


40 




Mastery Test (2) 



Problem 3 : Describe in detail some considerations which might alter the 

level of significance employed in testing hypotheses? Also, 
what do you understand to be the meaning of "veiel of signifi- 
cance? 



Problem 4 i Our sample has a mean of 67.47 and a standard deviation of 3.42 
and we wich to generate a 95 per cent confidence interval. We 
may assume that our distribution is symmetrical for our sample 
of 30 cases. Can we claim that this interval contains the 
true mean of the population? 



Problem 5 ; Using the same statistical data as in (4) above, but assuming 
that we have a skewed distribution in which we wish to place 
4 per cent to the left and 1 per cent to the right, compute the 
confidence interval and determine if this Interval contains 
the true mean of the population. 
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Mastery Test - Basic Correlation 



What is a correlation coefficient? 



The correlation coefficient (r) is the most commonly used measure of relationship between 
paired facts, or the tendency for two or more variables (or attributes) to go hand in hand. 
The quantity r, defined in Formula 1 below, is called the coefficient of linear correlate 
because of its theoretical relationship to a "straight line. 11 This coefficient is often 
called the "product moment" coefficient or the "Pearson r" statistic. 

When do v:e use correlation? 



In some problems our interest centers on the estimation of scores by means of a regression 
equation; other problems are concerned chiefly with the mutual interrelationship between 
two variables as measured by the coefficient of correlation. In the first case there 
would La practical value in predicting college achievement potential from a prognostic 
test — the relationship is known (from known scores or data) and forms the basis for 
estimating unknown scores. In the latter case - in correlation - the relationship between 
the variables is intuitively "known" but can also be readily expressed in quantitative 
form (e.g., r * .79). 

P roblem #1 - Calculation of Pearson "r" from ungrouped data : 

To find the coefficient of correlation on ungrouped data, we apply Formula 1^ and calculate 
six sets of component data — £Y., £X 2 j, 4.Y ^X^Y^, and N, and substitute the 

numerical values of these components into the formula. Although it is easier to employ 
a calculator or a computer when handling large number of cases, the coefficient can be 
computed manually. In order that you may gain a "feel" of the statistical operations 
accrued during the computation of "r", the following problem should be hand-calculatdd. 

Let us suppose that the values of X^ and Y^ which are given in the following table repre- 
sent the scores which 10 students obtained in two successive short quizzes in arithmetic 
and that we are interested in determining the relationship between the quizzes. Apply 
Formula 1 below to these data and compute the product -moment coefficient. (You will note 
that summation data for the necessary components are included at the base of the Table — 
check your computations against these data.) 



Formula 1 



_ - (£X i )(£Y i ) 

*y — ■ — ■ 

Vn£X 2 . - (£Xi) 2 VNlFJ - 




POOR ORIGINAL COPY -BEST 
available AT TIME filmed 
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TABLE 1 - DATA 



Student X Y X 2 Y 2 XY 



Mary 


8 


6 


Bill 


6 


7 


John 


10 


8 


Sue 


4 


5 


Carol 


8 


9 


Jack 


9 


10 


Irene 


3 


5 


Donald 


10 


9 


Bobby Jo 


6 


7 


Neil 


7 


4 



Totals £Xi= £Y i = £X 2 ± =555 £Y 2 i =526 £X i Y i =527 N= 



Substituting these component data into Formula 1^ we have : 



xy 



_ The computed correlation coefficient (r = ) described the strength of the association 

between the scores which students obtained on the two quizzes . A further interpretation 
of these data might indicate that 



You might find it helpful to plot the relationship between Test X and Test Y scores. The 
"r" computed above has a range limited to -1.00 (perfect negative relationship) through 

*0.00 (no relationship or chance) to +1.00 (perfect positive relationship. 

The four figures below illustrate the relationship between theoretical X and Y scores. 
Geometrically speaking a correlation is positive if the regression line slopes upward, and 
it is negative if the regression line slopes downward ( left- justified) . 



hi 


. <* *' 


hi 


hi 


4 

\4 


hi 


Y 


• * • 


Y 


• * Y 


• 

' K- 

V 


Y 


lo 




lo 


' + * lo 


•\- 

* 


lo 
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16 hi 

O X . , 

Positive r 



lo hi 

X 

Weak Positive r 



lo hi 

X 



\ 4 

' V . 

* N . 

• ' \ , 

* A ' 

* “\ • 



lo hi 

X 



Strong Negative r Weak Negative r 
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A rough plot of Table I data is as follows: 

(also plot a rough regression line) 

10 
9 

8 On the basis of the computed coefficient and its plot, 

7 we can conclude that: 

6 

Y 5 

4 

3 

2 

1 

0 

0123456789 10 
X 



The Rank-Order "r" : At times it is convenient to describe the correlation between paired 

observations by calculating r on the basis of their ranks instead of their actual numerical 
values. The advantage of this modification lies in the fact that it is much less laborious 
that the calculations by Pearson 1 s formula . To compute the correlation by ranks the fol- 
lowing formula may be applied: 

Formula 2: r = 1 - 6 1 

n(n 2 - 1) 

If we take, for example, the same data as presented in Table 1^, we can replace each value 
of X and Y by its corresponding rank within each test sample (administration). It does 
not matter whether we rank the Xs and Ys in ascending or decending order as_ long as we ace 
consistent. If we choose to rank the Xs from high-to-low, then we must also rank the Ys 
- from high-to-low. (For this problem, let us rank the Xs first, and in decending order) 



Student 


(X) Rank of Xs 


(Y) 


2 

Rank of Ys d (difference) d 


John 


(10) 


(8) 




Donald 


(10) 


(9) 




Jack 


(9) 


(10) 




Mary 


(8) 


(6) 




Carol 


(8) 


(9) 




Neil 


(7) 


(4) 




Bill 


(6) 


(7) 




Bobby Jo 


(6) 


(7) 




Sue 


(4) 


(5) 




Irene 


(3) 


1 5) 










id 2 .= 48.00 



Substituting the component data into Formula 2 above, we have: 



r = = 

r 

Compare the coefficients computed by the Pearson and rank-order methods. What can you 
say about these two statistics and, especially, when would you apply each method. What 
“might you be "giving up" by applying rank-order methodology rather than Pearson's method? 
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SECTION IV: SELECTED BIBLIOGRAPHIC REFERENCES 

This list of bibliographic references is not intended to be a com- 
plete listing of all of the possible sources pertaining to school program 
evaluation. Those references considered to be mr3t helpful to evaluation 
personnel have been selected for inclusion. The more basic discussions 
are preceded by asterisks. 

The references discuss some of the broad topic areas listed below. 

The letters "A 11 through ,f M M at the upper right-hand corner of each 
reference indicate the areas with which the works are primarily concerned. 
(For example. Bloom* s work Stability and Change in Human Characteristics 
contains some valuable comments about evaluation design and statistical 
analysis of data.) Although the letters indicate the primary foci of the 
references, the categories should not be thought of as completely 
independent. The classifications are as follows: 

A. Evaluation Design 

B. Evaluative Price dures 

C. Standardized Instruments 

I 

D. Informal Tests ; 

E. Preparing Objectives 

F. Statistical Analysis of Data 

G. Anecdotal Records 

H. Interviews 

J. Observation 

K. Ratings and Rankings 

L. Self-Report Techniques (Attitudes, Interests, Personality) 

M. Sociometric Techniques 
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Selected Bibliographic References 

B,D , G,H , J ,K ,L ,M 

1. *Ahmann, J. Stanley, and Glock, Marvin D. , Evaluating Pupil Growth 

(3d Ed.), Allyn (Boston), 1967. 

General textbook for elementary and secondary school 
teachers explaining the basic methods of evaluation 
and giving some basic definitions and statistical 
terms . 

A ,C 

2. Barrett, Thomas ( Ed • ) , Perspectives In Reading No. 8: The 

Evaluation of Children *s Reading Achievement , International 
Reading Association (Delaware), 1967. 

Careful analysis of the design and evaluation of reading 
programs. Includes much information about standardized 
tests. 

F 

3. *Bernstein, Allen L. , A Handbook of Statistics Solutions for the 

Behavioral Sciences , Holt (New York), 1964. Paperback. 

A,F 

4. Bloom, Benjamin S. , Stability and Change in Human Characteristics , 

Wiley (New York), 1964. 

Detailed and involved analysis of the development of 
human characteristics and how limitations such as 
environment produce a variation in the development. 

An attempt to understand how human characteristics 
may be identified, explained, and eventually modified. 

A 

5. *Borg, Walter, Educational Research: An Introduction , 

McKay (New York), 1963. 

Simple and practical presentation of educational 
research. Book presented in the order that 
researcher would use in developing and carrying 
out research. 

B,G ,D,G,L 

6. Bradfleld, James M. , and Mo*redock, Steward H., Measurement and 

E valuation in [education , Macmillan (New York), 1957. 

Introductory text for educators dealing with the basic 
concept of measurement and evaluation with some defini- 
tion of technical terminology. Also explained are the 
standards appropriate for evaluating pupil achievement 
and the most efficent way of reporting evaluation to 
pupils and parents. 
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D 

7. ^Bureau of Elementary and Secondary Educational Testing, Improving 

the Classroom Test , State Education Department (Albany), 1967. 
Pamphlet. 

C 

8. Buros, Oscar (Ed.), The 1940 Mental Measurements Yearbook , Mental 

Measurements Yearbook (New Jersey) , 1941. 

9. , The Third Mental Measurements Yearbook , Rutgers University 

Press (New Jersey), 1949. 

10. , The Fourth Mental Measurements Yearbook , Gryphon Press 

(New Jersey), 1953. 

11. , The Fifth Mental Measurements Yearbook , Gryphon Press 

(New Jersey), 1959. 

12. , The Sixth Mental Measurements Yearbook , Gryphon Press 

(New Jersey), 1965. 

Listings and critical analyses of standardized instruments. 
Instruments discussed in one edition do not necessarily 
reappear in later editions unless further data or critical 
reviews have become available. 



A 

13. Cochran, William G. , and Cox, Gertrude M,, Expe rim ental Designs 
(2d Ed.), Wiley (New York), 1957. 



C,D 

14. Cronbach, Lee J., Essentials of Psychological Testing , Harper (New 

York), 1960. 

Textbook explaining the basic principles of testing so that 
educators can select tests wisely, be aware of weaknesses in 
tests, and judge the merit of new tests emerging. Includes a 
discussion of some of the more important and widely used tests. 

A 

15. Davit z, Joel Robert, Evaluating Research Plans In Psychology and 

Education , Teachers College Press (New York), 1967. 

Guide to the establishment and evaluation of educationally 
oriented programs. 

D 

16. Dunn, S.S., Measurement and Evaluation in the Secondary School , 

Australian Council for Educational Research (Australia), 1967. 

Explanation of the* basic principles of testing with particular 
emphasis on the development of instruments suitable for 
testing on the secondary level. 
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C,D 

17. *J)uro 5 t, Walter N. , and Prescott, George A., Essentials of Measure- 

m ent for Teachers , Harcourt (New York), 1962. 

Presents the fundamentals of selecting, using, and interpreting 
standardized tests. This is a practical presentation of issues 
that arise in connection with tests and evaluation. 

C,D 

18. Ebel , Robert L., Measuring Educational Achievement , Prenti ce -Hall 

(New Jersey), 1965. 

Guide to the correct use of standardized instruments and the 
development of informal tests at the ocal level. 

F 

19. *Edwards, Allen L. , Statistical Ksthods in Behavioral Sciences , 

Holt (New York), 1954. 

Excellent text for research workers in behavioral science. 
Includes excellent treatment of two and three condition 
analyses of variance. 



20 . 

21 . 



Engelhart, Max D. , "Improving Classroom Testing, 11 National Education 
Association, What Research Says to the Teacher . 

A 

Federer, Walter T. , Experimental Design , Macmillan (New York), 1955. 



F,H,J 

22. Festinger, Leon, and Katz, Leon, Research Methods in Behavioral 
Sciences, Dryden (New York), 1953. 



23. 

24-. 



25. 



Early work that codified research techniques. Includes 
excellent chapter on Ehe collection of research data by 
interviewing and the method of sampling and analysis of 
data that can be useful i.o the individual planning an 
interview study. 

A 

Fisher, k.A., The Design of Experiments (2d Ed.), Oliver & Boyd 
(Edinburgh), 1942. 

F 

Fisher, R.A., Statistical Methods for Research Workers (6th Ed.), 
Oliver & Boyd (Edinburgh), 1936. 

C,F 

★Freeman, Frank S. , Theory and Pra ctice of Psychological Testing 
(3d Ed.), Holt (New York), 1962. 

Emphasis on the psychological foundations of tests and 
psychological interpretation and evaluation of test findings. 
Includes a chapter on the historical background of psychological 
testing and one on elementary statistical concepts. 
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A ,B, E ,F 

26. Gage, Nathaniel L. , Handbook of Resea rr a on Teaching , Rand McNally 
(New York), 1963. 

Many-authored handbook with the aim of summarizing, analyzing 
and integrating research on teaching into closer contact with 
the behavioral sciences. Intended as an aid in the training 
of workers in educational research. Excellent as a source 
reference for research in curriculum and classroom methodology 
and relevant variables. 



27. 



28 . 



F 

*Garrett, Henry E., Statistics in Psychology and Education , Longmans 
(New York), 1958. 

D 

Gerberich, J. Raymond, Speciman Objective Test Items , Longmans 
(New York), 1956. 



Guide for teachers explaining a wide variety of testing, 
measuring, and evaluating devices. 



C,D 

29. Gerberich, J. Raymond, et al ./ Measurement and Evaluation in the 
Modern School , McKay (New York), 1962. 

Supplementary volume presenting a short and nontechnical 
discussion of the application of measurement and evaluation 
to the problems of instruction in today's schools. Emphasis 
on improving homemade tests* 



~ " A 

30. *Good, Carter V., Introduction to Educational Research , Appleton 
(New York), 1963. 



Basic text giving a good description of the process of 
historical research and the major steps in carrying it out, 
including collection of data and the preparation of the report. 
Extensive updating has taken place in this edition, 

A ,H, J ,K 

31. Good, Carter V., et al., Methods of Research , Appleton (New York), 
1954. 

Excellent book developing educational lesearcli methods and 
techniques with a discussion of the concepts, principles, and 
procedures in educational, psychological, and sociological 
investigations. Organization of the book follows the steps 
taken in problem solving. 



B , G, J ,K ,L ,M 

32. *Gronlund, Norman E., Measurement and Evaluation in Teaching , 
Macmillan (New York), 1965. ~~ 
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F 

33 . Gui 1 ford , J • P . <> Fundamental Statistics in Psychology and education 
(3d Ed.), Me Craw (New York)," 1956. ” 

Fundamentals of descriptive and sampling statistics. This 
revised edition has materials on hypothesis testing, statisti- 
cal reference, and additional analysis of variance. 



34. Harris, Chester W. (Ed.), Encyclopedia of Educational Research 

(3d Ed.), Macmillan (New York), 1960. 

Includes writings appropriate to all aspects of evaluation 
and assessment. 

C,D,J,K,L,M 

35. Horrocks, John E« , Assessment of Behavior , Merrill (Columbus), 1964. 

Textbook designed for a course in psychological measurement.. 
Includes application, background, and underlying assumptions 
of measurement; materials on measurement of personality, 
maturation and readiness, intelligence, achievement, aptitude 
and special abilities, social behavior, and interests and 
attitudes* 

B,K 

36. Jahoda, Marie, et al • , Research Methods in Social Relations , Dryden 

(New York), 1951. 

Textbook dealing with all the steps in the research process 
used in the study of social relations. Book is divided into 
two parts: one deals consecutively with major steps of 

scientific inquiry; the other deals in more technical detail 
wtih methodological problems. 



M 

37. Jennings, Helen H., Soclometry in Group Relations (2d Ed.), American 
Council on Education (Washington), 1959. 



Revised and expanded volume on the meaning of sociometric 
structures and cholceB. 

\ 

A,B,>\H,J,K,L,M 

3B. Kerlinger, Fred N., Foundations of Behavioral Research , Holt 
(New York), 1965. 



A thorough treatment of techniques of educational and 
psychological inquiry, with special attention to the relation- 
ship between the problem and the dejlgn and methodology of its 
solution. Many practice! applications are presented. 



39. *Koenker, Robert H. , Simplified Statlsltlcs , McKnight (Illlnlos), 
1961. Paperback. 
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40. *Kowitz, Gerald T. , Research for Educational Improvement , State 
Education Department (Albany), 1960. Pamphlet. 



C,D,E,F,G,K 

41. *Lien, Arnold J. f Measurement and Evaluation of Learning; A Handbook 
for Teachers , Brown (Iowa), 1967. 

A thorough presentation of types of measurement and 
evaluation as they apply in the classroom setting. 



B,C,D,E 

42. Lindquist, E.F. (Ed.), Educa t i ona 1 Mca s u r omen t , American Council on 
Education (Washington, D.C.), 1963. 

An indispensible reference work and handbook to measurement 
specialists; test constructors, research workers in education 
and psychology, and school administrators in charge of local 
test programs. 



E 

43. *Mager, Robert F., Preparing Objectives for Programmed Instruction , 

Fearon (California), 1962. Paperback. 

Refreshing book for teachers, describing how to specify 
educational objectives « Provides both a valuable approach 
to the task of goal specification and an orientation which 
views goal specifications as an unavoidable, practical 
problem requiring hardheaded Solutions. 

A ,B 

44. *McAsban, Hildreth H. , Elements of Educational Research » McGraw 

(New York), 1963. 

Easily readable, introductory work on research techniques 
for use by teacher and education administrator. Practical 
book rather than theoretical. 

M 

45. *Northway, Mary, A Primer of Socioinetry , University of Toronto 

(Canada), 1952. 

Introductory work giving the basic principles and practices 
of sociomotry for the study of social relationships. A 
basic bibliography is given to guide the reader in this 
intricate and complex field. 



D,E 

46. *0dell, Charles W. P How to Improve Classroom Testing . Brown (Iowa), 
1953. 

Practical, nontechnical guide for teachers in the construction 
and administration of informal or homemade tests. Work based 
on the author^ four decades of teaching experience. 
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47. Payne, David A., and McMorris, Robert F. (Eds.), Educational and 
Psychological Measurement: Contributions to Theory and 

Practice , Ginn (Boston), 1967. Paperback. 

Book of readings presenting a broad perspective of 
theoretical and practical considerations in measurement. 



48. Rothney, John W.M. , "Evaluating and Reporting Pupil Progress," 
National Education Association, What Research Says to tin; 
Teacher. 



A,F 

49. *Sicgol, Sidney, Nonparametric Statistics for Behavioral Sciences , 
McGraw (New York), 1956. 



Presents nonparametric techniques in a form that can be 
understood by the average behavioral scientist */ho lacks 
advanced mathematical training. Emphasis on research 
application of techniques with many interesting examples 
taken from the behavioral sciences. 



E 

50. Smith, Eugene R. , and Tyler, Ralph W. , Appraising and Recording 
Student Progress , Harper (New York), 1942. 

Detailed report on the steps to be taken to help schools 
discover, record, and report the progress of students 
toward the whole range of goals. 



F 

51. *Smlth, G. Milton, A Simplified Guide to Statistics , Holt (New York), 
1962. Paperback. 



B,C,D 

52. Stanley, Julian C., Measurement in Todays School (4th Ed.), Prentice 
(New York), 1964. 

General reference book concerned with the theory and practice 
of measurement. Emphasis on the many problems relating to the 
intelligent use and interpretation of measurement by the class* 
room teacher and school administrator. 



53. 

54. 



F 

Tate, M. , Statistics in Education , Macmillan (New York), 1955. 

' ‘ D,E 

Taxonomy of Educational Objectives , Bloom, B.S., et al«, Longmans 
(New -York). 
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Handbook I: Cognitive Domain, 1956* 

Describes the nature of the cognitive domain and its relation- 
ship to educational goals* Includes a description of the stages 
of the hierarchical structure and relates educational objectives 
to testing* 

Handbook II: Affective Domain, 1964. 

Describes the nature of the affective domain and the classifica- 
tion structure prepared for it. Describes the evaluation of a 
affective objectives at each level of the structure* 

C,D,fi»F 

55* Thorndike, Robert L. , and Hagen, Elizabeth, Measurement and Evalua- 
tion in Psychology and Education (2d L), Wiley (New York), 

1962. 

Textbook with express aim to provide a foundation for persons 
who will use and interpret tests; defines the objectives of 
measurement and the value of teacher ^made tests. Includes a 
discussion of new tests. Easy-to-read coverage of educational 
and psychological measurement. 



56. Torrance, E. Paul, •'Crcativi ty, 11 National Education Association, 
What Research Says to the Teacher. 



A,H,J,K 

57. *Travcrs, Robert M.W., An Introduction to Educational Research , 
Macmillan (New York), 1958. 

Basic text discussing the aims and methods of educational 
research. Includes an explanation of how research should be 
conducted, the content and selection of research problem, 
some points of evaluation, and the validity of measurement. 



C,G,H,J,K,L 

58. Traxler, Arthur E. , Techniques of Cuidancc , Harper (New York), 1945. 

Practical guide giving the techniques and procedures available 
for the all-important job of counseling with explanation of 
tests and other instruments of evaluation. 



F 

59. Walker, Helen M. , and Lev, Joseph, Statistical Inference , Holt (New 
York) , 1953. 

Good reference book on statistical theory and methods for the 
non-mathematical ly oriented student • 
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60. Wandt- Edwin, A Cross-Section of iklucatlonal Research , McKay (Now 

Vork) , 1965. 

Collection of forty educational research articles published in 
journals in 1960 to 1964. The primary purpose is to acquaint 
administrators and teachers with recent research finding; of 
practical application. 

F 

61. Winer, B.J. , Statistical Principles in Experimental Design , Me Craw 

(New York), 1962. 

C,D,E,C,H,J,K,L,M 

62. Wrightstone, J. Wayne, Justman, Joseph, and Robbins, Irving, 

Evaluation in Modem Education , American Book Company 
(New York), 1956. 

Presents many of the problems which may arise in educational 
evaluation, suggests solutions. Includes samples of several 
types of assessment devices. 
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TOPIC: The Ingredients of the Research Proposal 

SESSION NUMBER: 18 



INSTRUCTOR: Robert L. Baker 

OBJECTIVES : 

1. Identify the basic components of a sound research proposal, 
irrespective of variation in proposal format. 

2. Describe the conditions that must be met for each component. 

3. Distinguish between exemplars and non-exemplars of proposal 
components based on conditions to be met. 

4. Given a problem area, construct a proposal outline including the 
operational specification of essential ingredients. 



CONTENT OUTLINE: 

I. Proposal Formulation and Planning 

II. Descriptior of Basic Components of a Sound Research Proposal 

A. Differences in labels attached to components in varied 
proposal formats. 

B. Differences in ways of sequencing and/or organizing 
components in overall proposal structure. 

III. The Conditions That Must Be Met for Each Basic Component 
of a Sound Proposal 




OBJECTIVE 6: Identify the essential components of a sound research proposal 

and describe operationally the specific conditions that each 
component must meet. 



The Increased financial support of educational research activities by govern- 
ment agencies and privately endowed foundations provides wider opportunity 
for the conduct of sorely needed educational research. The increase in 
funding, however, has been outpaced by the number of research proposals 
submitted for consideration by the funding sources . In a competitive research 
market, the weak, infirm, or poorly stated proposal is quickly cast aside. Thus, 
the educational researcher to work his trade must, in addition to his other 
skills, acquire competency In the development of a sound and well-explicated 
research proposal. Despite the vindications of the rejected researcher, most 
proposals are rejected not because of the researcher's lack of sophistication 
regarding the complex subtleties of "grantsmanship" , but rather because 
his proposal lacked one or more of the essential ingredients of a sound 
research proposal. 

The purpose of the sessions related to Objective 6 is to describe the 
essential ingredients or components of a research proposal and to identify 
the conditions that must be met for each component. The components can be 
identified in any good research proposal regardless of the variations of 
format. They can be easily identified in proposals which attach different 
labels to the components or specify different ways of sequencing and/or 
organizing the components in the overall proposal structure. 

The basic ingredients described in this session should be regarded as a 
necessary but not sufficient condition for the development of a sound and 
well-explicated research proposal. 



THE BASIC COMPONENTS OF THE RESEARCH PROPOSAL 



I. Proposal Formulation and Planning 



A. Formulation of research problem 



The first stage in the formulation of a research project is the 
selection of a fruitful problem that is significant for education. 
The range of potential problems or situations requiring change is as 
broad as the range of behaviors related to educational activity. 

The significance of a problem rests on its probable contribution 
to knowledge. The following are some of the important criteria 
which should be carefully considered in the selection and/or 
formulation of a research problem. 

Suggested Criteria for Selection of Research Problems* 

1. A concern xjith basic concepts and relationships of concepts 
as distinguished from local, particularized, or exclusively 
applied research, to the end that the knowledge produced may 
be cumulative with that from other studies. 

2. The development, refinement, and testing of theoretical for 
formulations, 

3. Superior research design, including careful specification of 
the variables involved and use of the most precise and 
appropriate methods available. 

4. A probable contribution to methodology by the discovery, 
development, or refinement of practicable tools, techniques, 
or methods. 

5. Full utilization of relevant concepts, theories, evidence, 
and techniques from related disciplines. 

6. The integratic of any single study in a planned program of 
related research to the end that the results become meaning- 
ful in a broad context. 

7. Adequate provision to train additional research scientists. 

8. Provision, wherever feasible, to repeat or check related 
research of other persons in order to provide a check on the 
generality of conclusions. 



*From Report of the Study for the Ford Foundation on Policy and 
Program ), Detroit, Michigan: Ford Foundation, November, 1949 




B. Planning the research proposal 



Planning of the research proposal represents the next step in the 
proposal-writing sequence • 

Before such planning begins, it is assumed that the researcher (1) 
has made the necessary canvass and review of relevant background 
literature and (2) that he has formulated the research problem and 
determined the sequence of operations and procedures for solving the 
problem. He is now ready to describe his research procedure and his 
findings and to correlate them with the findings of other researchers 
and with theory in his field. 

1. First considerations in planning a research proposal 

a. Determination of appropriate funding agencies for proposal 
research . 

b. Determination of selected funding agency’s research proposal 
format. 

c. Consideration of length and detail of proposal. 

d. Getting the overall problem clearly in mind (consideration 

of the major aspects of the problem in light of the hypotheses 
that will guide the research) . 

e. Consideration of what is directly and immediately pertinent 
(i.e.j what should be included in the proposal) and what is 
of secondary concern and should be omitted or placed in an 
appendix. 

2. Outlining — The Organizational Plan 
a. Outline form 

The outline for the research proposal shows (1) the sequence 
of topics as they will be covered and (2) relationships 
between the various topics. A good logical outline aids both 
the reading and the writing of the research proposal. 

In outlining, logical organization can usually be facilitated 
by coding the divisions of the outline. The two mo^t fre- 
quently employed codes for sections or categories of the out- 
lines are (1) the use of numbers and letters and (2) the use 
of a decimal system. These are illuminated on the following 
page. 




Numbers and Letters System 



Decimal System 



I. 

II. 

A, 

B. 

III. 

A. 

1 . 

2 . 

B. 

1 . 

a. 

b. 

2 . 

C. 

1 . 

a . 

( 1 ) 

( 2 ) 

b. 

2 . 

IV. 



1.0 

2.0 

2.1 

2.2 

3.0 

3.1 

3.11 

3.12 

3.2 

3.21 

3.211 

3.212 

3.22 

3.3 

3.31 

3.311 

3.3111 

3.3112 

3.312 

3.32 



Either a topic outline or a sentence outline may be employed, 
but in either case it is important to remember that labels or 
headings for each section must be clear and meaningful. 

b . Logical division 

Principles of logical division should be followed in outlining. 

Chief points to remember are: 

(1) .. Each class or topic included... in the outline, should be 

based on a single principle of differentiation. 

(2) Each division in the outline should be mutually exclusive. 

(3) Each class or division in the outline should be exhaustive. 

(4) Each class should be well defined. 

(5) There should be no further division of a class unless 
there are at least two subclasses. 

c. A suggestion for outlining a research report 

Educational researchers frequently find that the use of a topic 
card system facilitates the organization and integration of material 
for the research proposal. Each topic or item that seems suitable 
for inclusion in the proposal is written down on a separate card. 







A topic-card should be prepared for each point that is rele- 
vant regardless of whether it is of major or minor impor- 
tance. Following the preparation of the topic card, each 
card should be checked against the criteria for the basic 
components of the proposal to eliminate cards that are not 
pertinent and to prepare new cards for topics that may have 
been omitted or overlooked. 

Then, the topic cards may be classified and organized, follow- 
ing the rules of logical division — putting topics together 
that belong together, arranging the topics that are coordinate 
and those that are subordinate, and so on. In this process 
it may be convenient to spread the cards out on the floor and 
arrange the cards in outline form. New cards may be inserted 
where they seem to be necessary, or cards may be eliminated if 
they are not pertinent in the total picture. 

After an effective order has been arrived at, the cards may be 
coded (using either the number and letter code or the decimal code 
noted before) and the material copied to provide the working outline 

The procedure is a very flexible one, making it convenient to 
add cards, subtract them, combine topics, revise the order, etc. 

Basic Components of a Research Proposal 

A. Introduction 

A well-explicated and technically sound research proposal contains a 
number of basic components that can be easily identified independent 
of variations in proposal formats. Different funding sources may 
require that different labels be attached to these components or 
they may specify different ways of sequencing and/or organizing the 
components in the overall proposal structure. The following 
constitutes a listing of the basic components of sound proposals: 

1. Project Identification (Title) 

2. Statement of the problem to be solved or situation to be improved 

3. Justification for proposal approach 

4. Operational research objectives, hypotheses and/or questions 

5. Sequence of operations and procedures to be used in solving 
problem. 



6. Evaluation of data. 



B. Description of the basic components of a research proposal 

The elements or conditions to be met for each component includes 
the following: 

1* Proj ec t Identificatio n (Title) 

A good project title consists of a short concise statement 
containing the following elements: 

a. Identification of variables 

(1) Experimental study 

(a) Independent variable 

(b) Dependent variable 

(2) Correlational/status study 

(a) Related variable 

(b) Outcome 

b. Identification of target population 

c. Specification of type of relationship between variables 

(1) Functional = ;, the effects of" 

(2) Non-functional = "the relationship of" 

2. Statement of the problem to be solved or the situation to be 
improved 

This component of a sound research proposal contains the following 
elements : 

a. Specification of the problem 

(1) Clear, brief statement of the problem with concepts 
defined where necessary, including: 

(a) Identification of manipulabie instructional variables 
for determination of their differential effects. 

( All relevant variables should be considered. The 
failure to consider relevant variables is a common 
and serious error in development of a research 
proposal.) 

(b) Identification of dependent variable 
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(c) Identification of target population* 

(2) Problem is delimited to bounds amenable to treatment 
or test. 

(3) Description of the signifiaance of the problem for education 
with reference to one or more of the following criteria: 

(a) Is timely. 

(b) Relates to a practical problem. 

(c) Relates to a wide population 

(d) Relates to an influential or critical population. 

(e) Fills a research gap. 

(f) Permits generalization to broader instructional 
principles or general theory. 

(g) Sharpens the definition of an important concept ^r 
relationship. 

(h) Has many implications for a wide range o£ practical 
educational problems. 

(i) May create or improve an instrument for observing and 
analyzing data. 

(j) Provides opportunity for gathering data that is restricted 
by the limited time available for gathering particular 
data. 

(k) Provides possibility for a fruitful exploration with 
known techniques. 

As pointed out by Smith (1965), it is apparent from the Research 
Advisory Committee’s reaction to proposals, that it views some 
problems as central to the field of education and others as peripheral. 
Studies which explore the learning process, or which seek to identify 
factors associated with the retention of students in schools and colleges 
are regarded as having greater significance than, for example, studies 
of adjustment problems of young adults. The latter topic may be a 
valuable one for research in another context such as mental health. 

The committee, however, must make judgements only about the signif- 
icance of each problem to the field of education. 
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b . Specification of relationship to theoretical framework (if 

pertinent) 

(1) Describe the relationship of the problem to a theoretical 
framework. 

(2) Demonstrate the relationship of the problem to previous 
research * 

(3) Present alternate hypotheses considered feasible within 
the framework of the theory (strong inference) . 

3. Justification for propoeed approach 

An ixnpottant criterion in the evaluation of submitted proposals 
by funding agencies is the potential cost-effectiveness of a par- 
ticular proposed research project* Cost-effectiveness indicates 
that there is a relationship between the cost of the proposed 
project and the results that are likely to be produced by the study 
(i.e., the effectiveness of the project). The proposal will be compet- 
etiva with other proposals to the extent that the results which are 
likely to be produced are sufficient to justify the cost. 

4. Operational research objectives, hypotheses, and/or questions 

•Che objectives, questions and/or hypotheses presented in the research 
proposal represent the researcher’s attempt to focus his attention 
on specific aspects of a problem. As Smith (1963) points out, the 
sharper the focus, the greater the probability that the experimenter 
will succeed in his task. A statement of the objectives, hypotheses, 
or research question which is broad and ambiguous will ultimately 
lead to conclusions which are also broad and ambiguous. 

The following is an example of a statement of a hypothesis included, 
in a proposal submitted to the Cooperative Research Program. tf The 
broad hypothesis is that a procedure can be followed which will 
lead to the initial formulation, revision, and final development 
of a broadly conceived theory of education based upon psychological 
and other relevant research findings.” It is apparent that, stated 
this way, a research objective, question, or hypothesis will have 
little or no meaning. To assure that the proposal is presented in 
sharp forms, it is necessary to state the hypothesis, objectives or 
questions in operational terms- -that is, the procedures and/or 
behavioral outcomes must be clearly specified and observable. 

a. Operational definition 

The necessity for operational definitions in the formulation 
of the research objectives, hypotheses and/or questions has 




been long emphasized in the literature. An operational definition 
requires that the observable conditions necessary for a concept *5 
application be stated. The concept is thus defined by the set of 
operations linking it to the conditions of observation. One should 
be cautioned, however, against assuming that an operational definition 
can be taken as final. Wiilower (1963) points out that, although 
most psychologists agree that IQ provides an operational definition 
of intelligence, this argument should not result in a failure to 
devise newer and better measures of intelligence, not in a moritorium 
on development of new theories of intelligence. 

The researcher must also carefully assess the scope of his 
operational definitions since narrowness can often be the price of 
precision in the formulation of a research objective. Without 
question the property of objectivity, which is afforded by operational 
definitions, is an essential requirement of scientific research. 
However, operational measures which omit too much or perhaps something 
crucial, have limited value. Operational definitions > as pointed out 
by Finan (1962), when properly used are a tool of theoretical 
research. Their purpose is to make concepts unambiguous, not to make 
it possible to theorize without concepts. 

b. Objectives 

The objectives component of a proposal provides a frame of reference 
for the evaluation of the remainder of the proposal. In this section 
the research objectives must be specified in terms of observable 
behaviors to be performed by the learner. 

The objectives statement should include: (1) an operational specifi- 
cation of what individuals in the target population will be able to 
do upon attainment of the objectives; (2) a description of the condit- 
ions or situations in which the presence or absence of the behaviors 
can be observed and recorded; and (3) specification of the antici- 
pated or acceptable level of performance for the target group. To 
minimize a broad ambiguity and conceptual confusion, a non-overlapping, 
simple taxonomy of behavioral objectives should be used. 

c. Hypotheses 

(1) Form of hypotheses 

Hypotheses are relevant to theoretical research. Thus, when a 
hypothesis is stated, it is essential to provide a thorough explanation 
of the theoretical framework or basis that leads to the hypothesis. 

If the theoretical framework cannot be stated, it is not appropriate 
to propose hypotheses but rather to pose research questions. This is 
particularly true in the event that the inquiry is directed in an 
area in which the experimenter is relatively unsophisticated. 




Research hypotheses take either one of two forms? (1) the null or 
statistical hypothesis: (2) the alternative or experimental hypothe- 
sis, The hypotheses can be presented as Guba (1963) points out, in 
four different kinds of statement including: 

(a) Literary null: a i: no-dif f erence” or "'no-effect 11 of the 
hypothesis expressed in terms of theoretical constructs. 

(b) Operational null: a ,f no-dif f erence f: form of the hypothesis 
expressed in terms of the operations required to test the 
hypothesis . 

(c) Literary alternative: a form expressed in terms of theoretical 
constructs that state the hypothesis that will be accepted if 
the null hypothesis is rejected. 

(d) Operational alternative' a form expressed in terms of operations 
that state the hypothesis that will be accepted if the null 
hypothesis is rejected. 

The null hypothesis (or the statement of the hypothesis in a 
n no-difference n form) is the most commonly used mode of phrasing the 
hypothesis. This form of the hypothesis is semantically difficult, 
particularly if more than one independent variable is included in the 
study. Statisticians, however, prefer this form of the hypothesis 
because it accurately reflects the probablistic models underlying 
the statistical techniques. The alternative or experimental hypothesis 
is simply a statement of the hypothesis the researcher proposes to 
accept if the statistical hypothesis is rejected. The alternative 
hypothesis specifies the anticipated outcome, i.e., a non-chance 
occurrence. Either the null or alternative form may be used alone at 
the experimenter's discretion. However, as a matter of convention, 
the null hypothesis is customarily given with the accompanying 
alternative hypothesis. 

Whenever there is a basis for prediction in the study, hypotheses 
should be stated as succinct predictions of the anticipated outcomes 
and/or findings rather than in the null forgi. 

Both the null and the alternative hypotheses can be expressed in 
either of two language forms., i*e., literary or operational. If the 
language is in the literary form, the hypothesis will be expressed 
in terms of specific theoretical constructs. On the other hand, if 
the operational form is used, the hypothesis will be defined by the 
instruments used to measure the variables implied by a theoretical 
construct. In educational research, the operational form of the 
null and alternative hypothesis is preferable on two accounts: 

(1) it exposes the logic of operational techniques and measurement 
devices used in the study, (2) it minimizes ambiguity and conceptual 
confusion that frequently results in using the literary form alone 
without translation into operational terms . 




(2) Criteria for evaluation of hypotheses 

To assure the formulation of testable and significant hypotheses, 

the following criteria may be utilized: 

(a) The hypotheses must be clearly stated in operational 
terms . 

The null and alternate hypotheses should be clearly 
stated and include the operational specification of the 
concepts, independent and dependent variables and 
measuring devices employed in the study. 

(b) The hypotheses must be specific and testable. 

All the operations and predictions included in the 
hypotheses must be thoroughly and unambiguously defined 
in order to assess the testability of the hypotheses. 

As indicated by Goode and Hatt (1952), hypotheses are 
frequently stated in such general terms that they are 
simply not testable. By operationally specifying the 
hypotheses, the potential validity/ of the experimental 
results is increased. By using broad terms one is able to 
resort to selective evidence in the interpretation of 
the results. Although astrologists and palm readers 
make their living by stating predictions in such a form 
that almost any occurence can be construed as prophecy 
fulfillment, such a strategy is not legitimate in 
educational research. As Goode and Hatt (1952) point 
out, the more specific the prediction, the smaller the 
chance that the prediction will actually be borne out 
as a result of a mere accident. It is imperative, then, 
that the research hypotheses be as explicit and specific 
as possible in order to avoid the trap of selective 
evidence. 

(c) The hypotheses should be related to a body of theory. 

This criterion is one that is frequently overlooked in 
educational research. In curriculum research it is valid 
to select a research problem which does not relate 
directly to previous curriculum research or theoretical 
formulations. The researcher should remember, however, 
that often with careful development, the same research 
study may not only obtain the desired information related 
to local needs, but may help to refute, qualify, or 
support existing instructional theories. 




(d) Questions 



Questions are relevant to status and correlational 
studies (i.e., How many are there? Is there a rela- 
tionship between them?). The questions is also appropriate 
when, as is often true in educational research, the re- 
searcher cannot state the theoretical basis for the 
study. In this case, it is appropriate to raise questions 
rather than propose hypotheses . The same criteria of 
operational specificity apply to the starting of research 
questions as were mentioned in the discussion of ob- 
jectives and hypotheses. The experimenter indicates by 
the specificity of the research questions he raises, how 
carefully he has thought through his problems. A research 
question of the form, "What is the relationship of in- 
telligence to reading and arithmetic achievement? 11 is 
too broad and ambiguous. A better statement of the 
question would be, "‘.That is the relationship of Otis 
Quick Test Intelligence Scores to the SAT Arithmetic and 
Reading subtest scores of seventh grade public school 
students?” 

5* Sequence of operations and procedures to be used in solving problems 
a. Introduction 

This component of a research proposal is variously labeled 

method, procedure, or in one instance method of procedure. 

The basic function of this component is to describe the 

operations that will be performed to solve the provelm of 

concern, including: 

(1) Specification of the operational evaluation-revision 
procedures that will be employed in the experiment. Often 
this is done by specifying sub-objectives of the overall 
experiment and indicating the empirical criteria that will 
be used to determine the attainment of each. 

(2) Specify the sequence of steps to be taken toward the 
solution of the problem and a list of products that will 
be produced at the end of each step. A "product” may be 
a new workbook or a film; it may be a method of pupil- 
grouping, or the specifications of an instructional 
procedure, or a vocabulary list. 

(3) Describe the method and rationale for sampling. 

(4) Describe the method and rationale for treatment assignment. 

(5) Specify the kinds of data to be collected, the method for 
collecting, and the rationale for using it. 
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(6) Describe the experiment el design and the rationale for 
using it. 

(7) Tell how the data will be analysed. 

Design 

A common failing of many proposals is that they neglect or ignore 
some of the posed research questions or hypotheses. Thus, the 
first step of the researcher in selecting an appropriate design, 
is to check every proposed questions or hypothesis to be sure that 
it is covered by the design unde:: consideration. 

The researcher should be aware of all the possible threats to 
validity and should disclose in this section how and why the 
design selected constitutes an optimal design under the part- 
icular constraints imposed on the experiment (e.g., inability 
to randomly assign subjects to treatment, limited size of sample, 
etc.) 

Although one should avoid expediency as a justification for a 
weak design, the researcher must convincingly demonstrate that, 
if a "true" experimental design is not feasible, the alternative 
design offered represents an optimal compromise based on (1) 
the extreme difficulty and/or cost of using a true experimental 
design for the proposed study; (2) the control of the variables 
selected; and (3) the control of possible sources of error. 

A well-explicated presentation and justification of an experi- 
mental deisgn should include most of the following elements: 

(1) Specification of how each research question or hypothesis 
is covered by the proposed design. 

(2) Specification of how specific anticipated confounding 
variables and threats to validity are controlled by the 
proposed design including: 

(a) Identification of variables which design will control 
and how specifically design will control them. 

(b) If specific sources of error cannot be completely 
controlled by proposed design, submit rationale as 
to why they are not anticipated to constitute major 
threats to validity (e.g., if testing- treatment inter- 
action remains as a threat to validity, indicate why 
such a threat is anticipated to have negligible 
effects. Cite supporting pilot study data if avail- 
able. It is imperative that the researcher 




demonstrate his awareness of all possible courses of 
error by pointing them out. If the researcher does 
not point them out, he can rest assured that very 
likely the reviewer will do so. 

(3) Specification of design in statistical or logical terms. 

The use of statistical or logical terms in describing 
the selected design as Guba (1963) points out, will 
facilitate communication and the identification of 
possible sources of error in the experiment. 

(4) Types of inferences that may be made using proposed 
design 

As pointed out in earlier sessions, the type of study 
(i.e., experimental — non-experimental) determines the 
types of statements of casual inference that may be 
made from the data. The researcher should verify that 
the selected design will permit him to make the kinds 
of statements that he would like to make from the 
acquired data. 

c. Target population and sampling 

(1) Sampling plan 

The target population and sample must be carefully 
described so that the reviewer can make an assignment of 
the generalizability of the experimental findings. 

The problem of external validity, as discussed previously 
in the session on experimental design, relates to the 
problem of knowing the population to whom the findings 
are applicable. To validly generalize the experimental 
results to the target population requires that the 
sample have been drawn from that population using an 
appropriate probability sampling plan. 

As Guba (1963) points out, two discrete steps are 
involved: 

(a) Ranaom selection of subjects from the target popula- 
tion, i.e., all subjects incorporated in the sample 
must be selected at random from the same population. 
This target population must be precisely defined. 

(b) Random assignment of subjects to treatment. 
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The rationale for the selection of a given sampling plan 
must be carefully specified. If stratefied, area or 
cluster samples are used, the reason for selecting any of 
these plans must be made clear and reasonable to the 
reviewer . 

(2) Sample size 

The problem of sample size cannot be adequately treated 
here. It should be mentioned, however, that one can estimate 
the sample size required to obtain a apecified level of 
power if he has adequate pilot study data or information 
from previous research using the same measurement devices. 

(3) Specification of sampling procedures 

It is necessary that the specification of the sampling 
procedure include at least the following elements: 

(a) Description of target population, experimental and 
control samples 

(i) Specification of target population to which the 

research objectives, hypotheses, or questions are 
relevant . 

(ii) Specification of procedures or rationale for 
determining size of sample to be used 

(b) Specificatinn of method of drawing or selecting 
sample 

(i) Specification of relative costs of the various 

sizes and types of samples allowed by the theory 

(id.) Specification of relative importance of Type I 
Error and Type II Errors 

d. Data to be gathered and measurement devices to be employed 
(1) Date to be gathered 

At least the following information should be provided 
in this section: 

(a) Specification of plans for collection of required 
data, including an explicity statement describing 
field controls to be employed. The major concern 
here is generally that you will maintain equivalent 
situations for all groups (Guba, 1963) . 




(b) Specification of the anticipated time schedule 



(2) Measurement devices to be employed 

(a) Description of all measurement devices to be used 
in study. If no adequate instrument currently 
exists, then it is necessary to specify in detail 
the procedures that will be used to develop an 
appropriate psychometric instrument. 

(b) Description of measures of qualitative variables 
including citation of available reliability 

and validity data. 

(c) Specification of rationale for selection of measure- 
ment devices including: 

(i) Supporting evidence that the selected devices 
have the appropriate psychometric characteristics. 

(ii) Defense of operational definitions used. 

(d) Description of use to be made of pilot study or 
test run. 

5. Analysis and evaluation of the data 

The description of the data analysis and evaluation procedures 
should include the following elements : 

a. Indication of consistency of method of analysis with research 
objectives and design. 

b. Specification of method of analysis including: 

(1) Use of special analytical tools * computer, card-sorter, etc. 

(2) Use of graphic techniques 

(3) Specification of types of tables to be constructed 

(4) Specification of statistical and/or other analytical 
procedures' to be utilized. 

c. Indication of the type of statements that may be validly 
made from analyzed data. 
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THE INGREDIENTS OF A RESEARCH PROPOSAL 



Exercise 1 

A. Discrimination of Good Proposal Titles 

For each of the following project titles * indicate whether: 

(1) The variables are clearly specified. If so, identify the 
variable (independent, dependent, etc.) 

(2) The target population is identified. 

(3) The type of relationship between the variables is 
correctly specified. 

Examples : 

1. The effects of a specially planned mathematics program on pupil 
^achievement in eighth grade mathematics. 

2. Reading comprehension, abstract verbal reasoning, and computation 
as factors in arithmetic problem solving. 

3. An o.rclysis of the relationship of selected factors to the nature 
of the voluntary reading of adolescents. 

4. The dynamics of two variants of classroom alienation. 

5. The relationship between achievement and verbal communication of 
secondary school children. 

6. The effects of anxiety and intelligence on concept formation. 

7. A study of concept learning and generalization in children. 

8. The effects of four instructional procedures on free operant 
discrimination and discrimination reversal in retardates , 

9. An investigation of the effects of two schedules of reading 

instruction on manifest anxiety and behavior adjustment: A comp- 

arison of varied amounts of time devoted to reading instruction and 
their effects on level of manifest anxiety and school behavior ad- 
justment among fifth and sixth grade children in a public school 
setting. 
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10. The relationship of certain administrative factors to the number of 
academic courses pursued by the academically talented students in 
the 1960 graduating classes of the public secondary schools of 
Delaware . 

11. The relationship between socio-economic status and problem solving 

ability: a study of the influence of experience. 

12. The relationship between the perceived emotional climate of the home 
of college students and certain variables in their functioning 
related to self-concept and academic functioning. 
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Construction of a Proposal Title 

For each of the following examples, construct a project title that 

satisfies the criteria for a good title. 

1 . A decision regarding the adoption of the PSSC physics course of study 
will be made from results of an evaluation of the course. Four 
schools can be involved in the study but no manipulation of enrollment 
into classes will be allowed. Each school will have at least two 
classes in Physics. Two of the four teachers can teach both PSSC and 
traditional but the other two can teach only traditional physics . 

2. A retention study is to be made with students who took BSCS Biology. 
About 750 sophomores (approximately 86 from each of 9 schools) took 
BSCS and are still in attendance as juniors. Data are available on 
these students and an equal number of sophomores who took traditional 
biology. 

3. An investigator wishes to determine the effect of reading ,f My Six 
Convicts' 1 has upon freshmen Sociology majors at College X in terms 
of attitudes toward prisoners. He is able to use all the sections 
of Introductory Sociology offered in the first semester. He has an 
appropriate measure of attitudes toward prisoners . 

4. A community committee wishes to discover what effects double sessions 
have on the quality of education of children in one of the crowded 
schools of their district. The bond issue has failed and they know 
they must have the pupils on double session for at least a year, but 
would like relevant information for publicity for future bond issue 
elections. The district has regular achievement and I.Q. testing 
programs in it's schools. What experimental design might be employed 
to gather this information. 



Exercise 2 



Below you will find three sections of a research proposal. Read it. 
Project Title 

The effects of Use of Sequencing and Mastery Principles in Film Pre- 
paration on the Concept of Formation and Procedural Performance of High 
School Students, 

Problem - 

The potential of 8mm silent cartridged film has captured the imagination 
of educators and film producers alike. The number of commercially available 
cartridged films is increasing at an expoential rate, portending the in- 
creased utilization of the motion picture as an instructional medium. Since 
the low cost of 8mm eliminates the most serious obstacles to local production 
of instructional films, greatly expanded activity in this area may also be 
anticipated. 



To date this technology has developed almost completely independently 
of previous and concurrent development in the psychology of learning and the 
technology of instruction. Consider the terminology which has already become 
standard in 8mm usage. Though nearly all commercially produced 8mm cartridged 
films are called "single-concept films," the definition of the term concept 
(as inferred from the characteristics of these films) lacks the precision 
that the term has achieved in psychological experimentation. Just as film 
producers might profit from a study of the meaning of the term concept as it 
is employed by psychologists and educational researchers, so the burgeoning 
8mm field can profit from an application of principles of learning and instruc- 
tional technology to the production of its increasingly popular product. 

The present proposal seeks to extend the applicability of two of the dom- 
inant principles that have grown out of controlled experimentation in programmed 
instruction to the production of 8mm silent cartridged films. These are the 
principles of sequencing and mastery. (Silberman, H., Coulson, J., Melaragno, 

R. , and Newmarlt, G., 1964; Schutz, R.E., Baker, R.L., and Gerlach, V.S., 1964) 

The principle of sequencing has two parts: (1) every skill and subskill 

included in the program objectives should be explicitly covered by the program 
unless it exists in the student f s entry repertoire; and (2) any materials that 
do not contribute to the program objectives should be eliminated. 

The principle of mastery states that the student should be required to 
demonstrate mastery of each eomponent subskill before he is allowed to advance 
to new topics based on the earlier material. 

These two principles will be investigated using two distinctive types 
of films as vehicles: (1) films designed to achieve concept fromation (the 

development of a new concept as opposed to the use of a concept already with- 
in the student’s repertoire {Travers, 1963, p. 127}); (2) films designed to 
develop procidural performance (motor behavior as opposed to cognitive be- 
havior) . It would appear that characteristics inherent in the motion picture 
medium render it uniquely superior to the textual medium in accomplishing 
both of these types of instructional objectives. 
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Experimental studies have been inconclusive with respect 
to the variables that influence the instructional 
effectiveness of motion pictures. The findings do 
suggest that certain general aspects may have import- 
ant implications when preparing film sequences. However, 
a fine-grain analysis of the influencing variables is 
still necessary to provide the precisions required for 
effective instructional control via the motion picture. 

Maccoby and Sheffield (1958) studied the optimal 
distribution of demonstration and practice in learning 
a procedural skill. They found that practice immediately 
following short segments of demonstration os better than 
deferring practice until a larger section of demon- 
stration has been completed. The general aspects of the 
study are compelling; that is, there is some utility in 
breaking the sequence into segments and in providing per- 
iodic practice en route. However, specifications pro- 
viding for learner response are imprecise from an instruct 
ion?l technology point of view. When one considers that 
the "segments" of the demonstration film were 5 or 

0 minutes long, it is apparent that we are dealing 
with stimulus conditions which are extremely gross in 
the light of current instructional technology. A much 
finer- grain analysis of the skills and subskills 
included in the objective and the nature and sequence 
of instructional stimuli is required. 

Hargoj.ius and Sheffield (1961) studied the effect of 
different lengths of filmed demonstrations of complex 
behavior sequences before an active practice response 
v’as made. The treatments included: (1) short demonstration 

then practice; (2) longer demonstration, then practice; 

(3) complete demonstration, then practice; and (4) a 
transition from short demonstration, then practice, to 
long demonstration, then practice. Although inconclu- 
sive with respect to establishing specifications of 
demonstration length, the aspect of active responses 
on route appears to warrant more extensive investigation. 
Again, extending the principles of sequencing and 
mastery will contribute to the precision necesaary for 
specifying optimal demonstration lengths and organization. 

One method of breaking a lengthy serial mechanical- 
assembly task into units has been suggested by Margo 1 ius , 

1 ffiold , end Mr.ccoBy (1961) . They "define a ' "natural " 
v.'.it is being contextually similar, and further 
cizpho.r i::o practice .on the "natural" units rather than 
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on the whole- Although procedural cues for specifying contextual 
homogeneity were not established, the general strategy warrants 
further, more definitive investigation. 

The results of these experiments , as well as other similar 
studies (Michael and Maccoby, 1961), reflect the potential product- 
ivity of greater experimental activity in this area- The general 
findings of these studies suggest gross definitions of the instruct 
zonal conditions which accompany effective film instruction. 

Such findings provide little in the way of generalizable procedures 
Greater experimental control in the systematic application of 
empirically derived principles of learning should help us to get 
beneath the surface description of instructional conditions and 
specify the procedural cues for developing instructional strat- 
egies related to the development of 3mm films • 

Objective 

To investigate the application of the principles of sequencing 
and mastery to the preparation of 8mm silent films designed to 
achieve concept format! - and procedural performance objectives. 

Now answer the questions below, using the following procedure. 

If the answer is yes , state your reason or else underline (in the 
sections above) the words on which your answer is based and place 
the number of the question in the left margin, next to the wotds 
you have underlined • If your answer is no, either state why or 
rewrite the pertinent portion of the proposal to meet the criteria 
implied in the question - 

1. Are the variables and type of relationship between the 
variables described? If yes, identify the variables (e.g., 
independent, dependent) , type of relationship between the 
variables - 

2. Indicate whether this would be a correlational, status or 
exp er iment al study. 

3. Is the situation requiring change described? If yes, specify 
briefly. 

A. Is the target population described? If yes, specify briefly- 

5. Specify what, if any, observable criteria will be used to 
determine when the desired change has taken place - 

6. Is the problem related to a theoretical framework? If so, 
specify. 

7- Are big definitions included? 
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8. Is the objective in the example stated in operational terms? 
If not, how specifically might it be modified so that it 
does comprise an operational statement. 

9. Is the problem of general, or opposed to local significance? 




Exercise 3 



Construct Proposal Outline 



Using one of the following problem areas or a problem area of 
special interest to you, construct a proposal outline using all of 
the criteria for a well-explicated proposal. 

1. A doctoral candidate is interested in the Zuni Indican culture. 
What he would like to do is to make a study of an Indian child of 
elementary s.hool age. In this study he specifically intends to 
investigate the child’s family background — habits, mores, education, 
attitudes — as it impinges upon this student. By studying the child 
and its background, this doctoral candidate hopes to learn a great 
deal not only about the ways and means for the elementary school 

to educate the Indian child. 

2. Your* school district has been using a team teaching approach 

in ninth grade English for three years* in each of eight high schools. 
The superintendent wished to know whether team teaching results 
in any greater learning than a traditional approach. He makes the 
resources of the Research Director available to you to use ninth 
grade r'"/ .** rvts in a nearby high school. These students are being 
taught ninth grade English by the traditional approach. The exper- 
iment is to run during the school year 1964-65. 

3. An investigator wants to see if the development of personality 
problems in first graders. A large school district with 10 elem- 
entary schools, each having the first four grades, is willing to 
coop erato. Fra-school reading readiness tests are administered 
each year and first grade classes are made up heterogeneously 

on the basis of these scores — Investigator free to work within 
this framework . 

4. The senior students in high school were net doing as well as 
some people thought the students should be doing on their compo- 
sition tests . It was decided to increase grammar content in the 
English curriculum. Someone questioned the reason for doing this. 
Therefore a study was planned to find out if increasing the 
grammar content would have any effect on composition grades. To 
aid the teachers engaged in the study all the senior English 
classes will be offered the same period next year. What design 
would you use? 
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TESTING HYPOTHESES: A PLEA FOR 



ORDER IN EDUCATIONAL RESEARCH 
Joe L. Saupe 

Michigan State University 

Recently, as I was making my first try at teaching a course in educational 
statistics , it became clear that in the process of confusing my students I had 
suceeded in confusing myself regarding the proper roles of problems, research 
hypotheses, and statistical hypotheses in educarioral research. My pet peeve 
regarding research in education, or is any other of the social sciences for that 
matter, is that all too often, research reports make too much of the null 
hypotheses. The epitomy of this glorification of the null hypothesis is its 
statement in the first chapter of a doctoral dissertation. I suspect that this 
over-emphasis of the null hypothesis results from the fact that either the student, 
his major professor, or both, once took a course in statistics — a course of 
action which I cannot criticize in itself. Unfortunately, instructors of 
statistics courses and authors of statistics textbooks are usually more concerned 
with the logic of statistics, in which the null hypothesis is rightfully central, 
than with the logic of using statistics in educational research. 

This may be a minor point to some. Where the null hypothesis is stated may 
not affect the data analysis or possibly even the conclusion of the research. 

But it is a major point with me. The logic of the use of the techniques of 
statistical inference is involved. 

In my feal to differentiate between research and statistical hypotheses for my 
students and to point out that null hypotheses are only statistical devices 
which permit the "proving" of research hypotheses, I emphasized that the research 
hypothesis was the one to be stated in Chapter I. Eventually, a Thoughtful 
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student asked, "But what if we don’t have any basis for developing a research 
hypothesis to state in Chapter I?” At this point I backed into a corner and 
mumbled something about a problem as opposed to a hypothesis. It is from this 
corner and the perspective it gave me that I attempted to think through the 
issues involved in the relation of statistical inference to educational research. 

I am sure that what I have to say has been said many times and many ways, but 
I want to say it again in my way in hope that this way will contribute to an 
orderliness in the logic involved in educational research and its reporting. 

Hypotheses and Problems 

I would like to suggest that basically there are two types of research for which 
the tools of statistical inference may be useful. The first I shall call Hypothesis 
Research, which, of course, invloves a problem in a larger sense of the word. If, 
in an area of investigation, there is a backlog of knowledge or tested theory 
which when slightly extended suggests a hypothesis for further investigation, or 
if the researcher develops or borrows what is, to him at least, a reasonable theory 
suggestive of one or more hypotheses, we have a basis for hypothesis research. 

Note carefully that such hypotheses usually (I’ll comment od the exception later) 
take the form of some positive statement about the relationships among a set 
of variables defined with regard to some population of individuals, objects, or 
events. Examples of research hypothesis are: "Students learn more under teaching 
method A than under teaching method B," "Self -perception is related to school 
achievement," "Authoritarian students are more likely to be school leaders than 
non-authoritarian ones," "Schools with democratic administrations are more likely 
to be adaptable than schools with authoritarian or laissez faire administrations." 
VJhile these hypotheses are not likely to be acceptable ones for thesis research 
because they lack the necessary qualifications and operational definitions, they 
should make the point. In hypotheis-type research, these are the hypotheses 
which should guide the research and which, therefore, should be stated at the 
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outset of the research report or dissertation in conjunction with the presentation 
or review of the theory on which they are based. 

1 

Two characteristics of such research hypotheses are worthy of note. First, 
the word "significant” does not appear in them. A research hypothesis that 
states, "There is a significant difference...," is really beside the point. It 
confounds the actual research hypothesis with the statistical hypothesis and thereby 
confuses the issue of using statistical inference as a tool of research. That 
"there is a signaficant difference...." is an acceptable, though not completely 
articulate was of stating a conclusion, but it never provides a meaningful state- 
ment of a hypothesis. 

The second characteristic of research hypotheses is that they usually specify 
a direction. If a body of theory or past research is the basis for a hypothesis, 
then the specific nature of the difference or relationship involved may usually 
be specified. Which of two teaching methods should prove superior is indicated 
or the sign of a relevant correlation specified. Ordinarily, if the theory does 
not suggest a direction for the hypothesis, the theory is incomplete regarding 
the hypothesis and only a problem (see below) exists. For thoses who are familiar 
with the terminology of statistics , I am saying that most research hypotheses 
are directional and hence call for "one-sided" or "one-tailed" statistical tests, 
but I am getting ahead of myself. 

The second type of research for which the methods of statistical inference are 
apprpriate I will call problem research. If there is no theoretical scheme which 
applies to an area of investigation or if the topic for study represents a sizable 
jump from such a body of theory, the researcher may then have no specific hypothesis 
concering the subject of his research. He has only a problem. He has no basis 
for guessing that there is or is not a difference between treatments or groups or 
relationship among the variables he chooses to study. It would then be out of place 
O 
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for him to attempt to formulate a hypothesis about the paticular piece of 
nature with which he is concerned. It is proper, of course, for him to state 
problems for investigation. Such problems may merely be declarations of what 
he intends to do. For example, he may set out ,! To study the responses of high 
ability students to personality measure X" or "To compare the behavior of males 
and females on vatiables Y and Z." Problems may also be stated as questions: 

"What are the attitudes of the voters of school system W regarding the place of 
vocational training in the school T s curriculum?" or "What is the relationship 
among certain socio-economic variables and students 1 self-concepts?" or "Which 
is a better predictor of academic success, Test A or Test B?" 

In problem- type research, the problem is the thing that guides the research 
effort. The problem, then, and not some artificial hypothesis should be 
clearly stated at the outset of the research report or in Chapter I of the 
dissertation. Research hypotheses have no logical place, no matter how they are 
stated, "null" or otherwise, in the formulation of this type of research. 

I am saying, in other words, that hypotheses are not absolutely necessary for 
research, even doctoral research, to be respectable. VJhile it is possible to 
argue that the significance (not statistical) of educational research can be 
measured by the extent to which it draws upon past research and a body of theory 
and contributes to it, it is still true that there is much virgin territory to 
be explored by educational researchers. The first study in an area cannot be 
guided by research hypotheses for further research efforts and thereby stimulate 
the development of a body of theory. Furthermore, some problem researchers may 
happen Gn to, by accident or design, relationships which suggest how some previously 
un considered theoretical scheme applies to the problem area or which provide for 
a basis of choice among possibly competing theories. 
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On the other hand, it is true that any hypothesis research is more likely 
(a statistical concept) to produce dividends on the store of knowledge than are 
many researchers of the problem variety- Some problems studied by educational 
researchers may have practical utility, but may lead nowhere in terms of a con- 
tribution to knowledge- Others may lead nowhere in any sense, but this consideration 
will not alter the proper role of statistical inference in their pursuance, 
which is after all the topic I have chosen to discuss. 

Statistical Hypothesis 

Statistics is an applied branch of mathematics . It has provided a body of 
knowledge and techniques which have been found quite useful in edicational research 
in making inferences — based on probability and random samples — concerning 
the populations from which the random samples were taken. The statistical 

1 

hypothesis is a central part of the techniques of this branch of knowledge. 

In essence, the method involves the positing of some hypothesis (statistical) 
about the nature of the population from which a sample is to be drawn. (Clearly, 
the word "significant 11 does not belong in a hypothesis vyhich is some positive 
statement about the nature of a population.) The sample is then drawn and observed. 
If it turns out to be quite atypical in terms of what samples from this population 
are expected to be like, the original hypothesis (statistical) may be subjected 
to doubt. If it is atypical enough, say belongs to that group of samples which 
are so unusual that we would expect them to be randomly drawn from the posited 
population only five times in one hundred samples, then we may say that we will 
reject the statistical hypothesis. If we play the game according to the rules, 
of course, we will state before we actually draw the sample how atypical it must 
be in order for us to reject the original statistical hypothesis. This is a prior i 
1 

While I am talking about the statistical testing of hypotheses, I do not mean to 
depreciate the value of statistical estimation which is an important statistical 
method, t. The logic of estimation is very similar to the logic of hypothesis testing. 
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definition of atypicalness is the matter of significance level and is expressed 
as a proportion, ,05, ,01, or ,001, We say that our observed sample must be 
among the fifty or ten or one most unusual that would be expected to occur 
in drawing a thousand of them before we will reject the original hypothesis 
about the nature of the population. 

The rejection of a a atistical hypothesis on the basis of an atypical random 

sample means that we are willing to behave as though the logical opposite of 

that hypothesis were true; that is, that the population is in fact of a nature 

different from the one we originally posited. In other vjords , we choose to 

believe that the sample is typical of, hence came from a population different 

2 

from that originally posited. We make this decision with a predetermined 
statement of risk; we know that in using this logic for rejecting stati tical 
hypothesis we will be wrong five or one times in one hundred, for even the 
most atypical sample could conceivably come from the specified population. 

Now, and this is important, the drawing of a "typical" sample which leads to 
the acceptance of the statistical hypothesis is not a logically sound basis 
for concluding that the specific statistical hypothesis we choose to examine 
is in fact true . The reason for this is that such a sample may actually be 
even more typical for any of a number of different populations which are very 
similiar to the one we elected to posit. In other words, the rejection of a 
statistical hypothesis is a stronger action than its acceptance. 

2 — 

Modern statisticians have built their theories and techniques around 
"alternative" as well as "statistical” hypotheses about populations. This 
refinement is important in modern statistical theory but may be ignored for 
the simple purpose of this paper. 
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Now we can relate statistical and research hypotheses, If a researcher has 
a research hypothesis which he wants to "prove*,’ his best strategy is to frame 
a statistical hypothesis which is its logical opposite and with all the 
objectivity, caution, and scientific detatchment to "disprove" that statistical 
hypothesis, (The words "prove” and "disprove" are put in quotation marks to 
indicate that they are used only in a probability sense,) Thus, when it 
comes time to analyze the results of his experiment , the researcher who 
developed the research hypothesis that students learn more under teaching 
method A than under teaching method B is wise to test the statistical 
hypothesis that teaching method B produces the same or more learning than 
teaching method A l If his results are sufficiently alien to this statistical 
hypothesis, he may conclude with his predetermined level of risk that teaching 
method A is superior to teaching method B. Q.E.D, That*s what he wanted 
to show in the first place. 

In problem research* the use of the statistical hypothesis is slightly 
different. Here the researcher merely selects a statistical hypothesis that 
appears to be relevant to his problem. Fortunately, in most cases, if the 
problem is defined with an adequate degree of specificity, he will find that 
there is really only one statistical hypothesis which is relevant to it. The 
teaching methods problem may be taken as an example. If there is no research 
hypothesis, the problem is to see if there is any difference between the 
effectiveness of methods A and B, The statistical hypothesis for this case 
is the "null" one that the two methods are equally effective. If our results 
are sufficiently inconsistent with this hypothesis, we reject it and conclude 
that they differ in effectiveness. 
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Under these circumstances, even though the average achievement score for method 
A were 100 and for method B 20, we could, strictly speaking, conclude only 
that the two methods differed in ef f ectiveness . We should then adopt the 
research hypothesis that method A is superior to method B and proceed to plan 
another experiment. 

Hypothesis research thus generally leads to somewhat more precise conclusions 
than does problem research. In statistical terminology, one-tailed or 
directional tests of hypotheses provide more precise information than do 
two-tailed or non-directional testp. The choice between a one-tailed and a 
two-tailed test of a hypothesis does not depend upon the general level of 
caution or scientific detatchment of the researcher, bat rather upon whether 
the nature of his research permits him to develop a (one-tailed) research 
hypothesis or a (two-tailed) problem* 

The point of all this is that the statistical hypothesis comes into the 
picture with the statistical methodology. It is an integral part of this 
methodology which permits research hypotheses or problems to be examined in 
the most efficient manner. Thus the statistical or null hypothesis should be 
relegated to the data analysis section of the research report or to Chapter 111 
or IV, depending upon the form required by the advisor of the dissertation- 
As a matter of fact, the precise statistical hypothesis can just about always be 
inferred from the nature of the specific statistical procedures employed in 
the data analysis. For this reason, I am not particularly disturbed if the 
research report does not include a precise statement of the statistical hypothesis 
being tested. 
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Purists in educational research may, however, rightfully require that the 
researcher demonstrate that he understands what he is doing in applying the 
logic of hypothesis testing by including in the research report a precise 
statement of the statistical hypothesis. 

SOME DIFFICULTIES 

Unfortunately, the techniques of statistics permit only certain types of 
statistical hypotheses to be examined or tested. Allowable statistical 
hypotheses may be roughly classified in two categories. The first of these 
includes the common "null ,* 1 no-difference, or no-relationship hypotheses. 

The other includes the difference-or relationship-in-one-direction types. 

It is clear from the foregoing section, then, that only two general types of 
research hypotheses or problems may be efficiently subjected to statistical 
examination. These logical opposites of the permissible statistical hypotheses 
are the difference- or relationship-in-either-direction situations (commonly 
the basis of problem research) and the difference- or relationship -in-the- 
other-direction situation (usually the basis of hypothesis research.) (In 
other words, statistical methods do not, as a rule, permit efficient examination 
of research hypotheses which are of the no-difference or no-relationship type.) 

But there are some situations in which the proper research hypothesis may 
take the "null 11 form. The best example is the case of ^testing assumptions’* 
as a prerequisite for applying some other statistical technique. In the case 
of the ,T t-test n of the null hypothesis that two population means are the same 
or that one is the same or larger than the other, it is helpful if it is 
reasonable to assume that the two (normal) population variances are the same. 

O 
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This research hypothesis is then a "null 11 hypothesis. Our only course of 
action in this situation is to test this same statistical hypothesis* If our 
sample data are sufficiently atypical — one sample variance sufficiently larger 
than the other — we must certainly reject that statistical hypothesis and 
behave, in applying the t-test, as though the population variances do in fact 
differ. On the other hand. If our data lead us to accept the hypothesis of 
no difference, we haven T t really ,T proven” much of anything. Recall that 
accepting a statistical hypothesis is not a very strong decision. (The point 
is that in this "case we can’t "prove” what we want to ’’prove”, namely, the 
null, research hypothesis. All we can say is that our data do not contradict 
that hypothesis, but this is not the same as saying that it supports it.) 

There are other types of assumptions of the no-difference or no-relationship 
type which are required in the use of various statistical techniques* For none 
of these assumptions is there a very conclusive statistical procedure available. 
In practice, however, it may not matter, because in many situations it doesn T t 
make too much difference if the assumptions are violated a little. If the 
population variances differ a little, the only effect of using the common 
t-test is that the significance level will not really be what we assume it will 
be. It may really be .06 instead of .05 and in a practical situation this is 
not a serious matter. 

My advice then is that researchers continue to test assumptions in the usual, 
and only available, manner but that they recognize that in accepting the 
null hypothesis they are providing only a very crude conclusion. If the popu- 



O 
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lation differences are sufficiently large that the sample data lead to a re- 
jection of the null hypothesis, they should not, as a rule, apply any statis- 
tical technique that is based on the assumption tested, because in this case 
they can infer that the significance level adopted for that technique may vary 
considerably from the one they would assume for it. 

Another difficulty in the use of statistical techniques in educational re- 
search is that some of the techniques are appropriate only for the testing of 
null, statistical hypotheses. Directional hypotheses cannot, strictly speaking, 
be studied with these techniques. While such techniques are useful for problem 
research, their value for hypothesis research is limited. Two specific tech- 
niques may be mentioned as examples. First, the chi-square test of independence 
in contingency tables is sensitive to any source of departure from independence 
in the data, but cannot (except in the one degree of freedom case) appropriately 
be used to "prove * 1 any specific source of dependence or any specific directional 
relationship. Similarly, the common analysis of variance method enables us to 
examine the statistical hypothesis that a set of population means are the same, 
but does not permit any very proper conclusions regarding which of the popula- 
tion means may be larger or smaller. To be sure, there are refinements and 
modifications of the fundamental analysis of variance procedures that are de- 
signed to permit directional inferences concerning population means, but these 
methods are approximations at best and do not alter the fact that the analysis 
of variance is basically a tool of problem research. 
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Summary 

The point of what I have been saying can be summarized as follows: Positively 

stated research hypotheses or specific problems for investigation should be 
developed and stated by researchers in the early sections of research reports 
or theses. These hypotheses or problems are, after all, the things which 
provide guidance to the research. Furthermore, there is no reason for the re- 
searcher to try to state a hypothesis, if in fact he only has a problem to 
study. Problems are perfectly respectable concerns for research. The statis- 
tical hypothesis, if stated precisely at all, should be reserved for the data 
analysis section or chapter of the report. It is an integral part of the 
statistical tool and should not be separated from this tool. Elevating the 
"null" hypothesis to the outset of the report tends to place undue attention on 
the statistical techniques of the study. These techniques are merely tools and 
should not be allowed to overshadow the substance of the research. 



ERIC 

ijfflinaffamiaaa 



COMPARISON OF EXPERIMENTAL AND EVALUATION STRATEGIES 





EXPERIMENTATION 


EVALUATION 


PURPOSE 


To test research hypotheses 


To facilitate the continual improve- 
ment of a program 


SUBJECTS 


Units to be measured are 
randomly assigned to treat- 
ment and control conditions 


Subjects are assigned to a program 
based upon their needs and the pur- 
pose of -the program rather than the 
requirements of the data collection 
and analysis designs 


CONTROL 


Treatment and control condi- 
tions are held constant 
throughout the experiment 


Evaluation aims to stimulate improve- 
ment in on-going pregrams 


INSTRUMENT 

ADMINISTRA- 

TION 


Instruments are administered 
after a specified period of 
time — usually a year or some- 
times pre and post to the 
experiment 


Instruments are administered to con- 
form with information requirements of 
decision makers throughout the pro- 
gram's existence 


^EDBACK 


Avoided during the experiment 
so as to avoid contamination 


An essential means for stimulating 
change 


CRITERIA 


Information should be: valid, 

and reliable 


Information should be: valid, reliable 

timely, credible and pervasive 



DEFINITION - 1 



Definition of MEANS vary according to the EIDS 

to fca served. 

Evaluation is a means to several ends: 

JUDGMENT 
DECISION-MAKING 
KNOWLEDGE OF RESULTS 
PUBLICITY 



* * * * * * * * * * * * * ^ 



DEFINITION - 2 



E VALUATION FOR DECISION-MAKING 

THESIS: A major purpose of evaluation is to provide information 

for making decisions. 
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RATIONALE : 

1. Quality education demands continuing efforts 
to improve education; 

2. Educational improvement requires an appropriate 
balance of enlightened persistance and change; 

3. Obtaining and maintaining this delicate balance 
requires sound decision making; 

4. Sound decision-making depends upon an appropri- 
ate supply of evaluative information. 



* * 



* * * * * * k * * 



DEFINITION - 3 

DECISION-MAKING IN EDUCATIONAL IMPROVEMENT 



PROBLEM: To evaluate for decision-making, the relevant 

decisions must be known. 

POSTUIATE: Decisions in educational improvement activities 

may be classified as: 

Planning (focusing needed improvement activities) 

Programming (specifying procedure, personnel, facil- 
ities, budget, and time requirements for 
implementing planned activities) 

Implementing (directing programed activities) 

Recycling (terminating, continuing, evolving, or 
drastically modifying activities) 



iV k k * * k * * * * * k k k k k * * 

DEFINITION - 4 

KINDS OF EVALUATION FOR DECISION-MAKING 

PROPOSITION: Each class of decisions in educational improvement 

activities requires a relevant kind of evaluation. 

Context evaluation is for planning decisions and provides 
information about "what needs to be done?" 

Input evaluation is for programming decisions and provides 
information about "what can be done?" 

Process evaluation is for implementing decisions and provides 
information about "what is being done?" 

Product evaluation is for recycling decisions and provides 
information about "what has been done?" 
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DEFINITION - 5 

FUNCTIONS IIT THE EVALUATION PROCESS 
All kinds of evaluation include four functions: 
Collection of information 
Organization of information 
Analysis of information 
Reporting of information 



DEFINITION - 6 



CRITERIA FOR ASSESSING THE ADEQUACY OF EVALUATIONS 

FOR DECISION-HARING 

Validity (Is the information what the decision-maker needs) 

Reliability (Is the information reproducible) 

Timeliness (Is the information available when the decision- 
maker needs it?) 

Pervas iveness (Does the information reach all decision-makers 
who need it?) 

Credibility (Is the information trusted by the decision-maker 
and those he must serve?) 



* ;V * 



* * 



* * 



DEFINITION - 7 




EVALUATION FOR DECISION-MAKING DEFINED 

EVALUATION FOR DECISION-MAKING is the process of: 

collecting , 
organizing , 
analyzing , and 
reporting information about 
contexts , 
inputs , 

processes, and 

products for decisions associated with 
planning, 
programming, 
implementing , and 
recycling activities. 
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STAGES IN THE CIPP EVALUATION MODEL 



1. Context Evaluation - defines and analyzes the environment, its 
unmet needs, and underlying problems. 



2. Input Evaluation - identifies and examines available resources, 
strategies and designs to meet program goals and objectives. 



3. Process Evaluation - identifies and monitors the potential sources 
of failure in a project. 



4. Product Evaluation - relates outcomes to objectives and to con- 
text, input, and process. 




HEW INSTITUTE 
ON 

EDUCATIONAL RESEARCH INNOVATIONS 

University of New Hampshire 
Summer 19G3 



NAME : PROPOSAL: Pre ; Post 

POSITION: INSTITUTION /AGENCY : 



REPORT AND ANALYSIS OF AN EVALUATION STUDY 



In your institute application you indicated briefly on a line or two 
what your current evaluation activities or responsibilities were -- such 
as “responsible for evaluating Title III programs", etc. We now want to 
know a good deal more about what you are really doing, the evaluation 
problems you face, and the ways in which you think about evaluation. 

We are asking for this fuller description for several reasons. First, 
it will give us a much better insight into the range and scope of your 
evaluation activities, so that we can adapt our institute program in 
various ways to deal more directly with your problems. Second, we can 
provide for you, shortly after our opening session, an informative pic- 
ture of who your fellow participants are, and allow you to initiate 
communication with other participants who may be facing similar evalua- 
tion problems. Third, it may help each of you to clarify some of your 
thinking about evaluation and your purposes for attending the institute. 
And finally, it will provide a useful baseline against which both you 
and we can estimate changes in ycur thinking and plans which may have 
been stimulated by your participation in the program. 

At the top of each of the following pages there is a question. The rest 
of the page is blank. Please "fill up" as much of the page as you can 
with your answer, and continue on the back of the page if you need to! 

So as to avoid unnecessary duplication, we suggest that you look at all. 
the questions before starting to write your answers. 

Please complete and return this form as soon as possible. We expect 
that you will get it to us nc later than Wednesday, July 3rd. Those of 
us on the institute staff regard this project as your first “assignment" 
and we look forward with great interest to the information and insight 
which your reply will give us. Also, we hope that it will be a worth- 
while and meaningful opening project: for you and "gear you" to the 
intent of the institute. 

Finally, please work out this assignment on your own as we'd like to 
know where you feel you are in the area of research and evaluation. Only 
then can we attempt to structure the body of the institute and also 
individualize our contributions to your evaluation activities. 
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1 . 



Describe in detail 
in evaluating (or a 
its nature, content 



a program you are now evaluating or have 
program to be implemented this year). 

» scope, purposes, etc. 



some part 
Indicate 
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2. What sc~:ts of data (Information) are you collecting (have collected 
or will collect) and how do they relate to the purposes of your 
evaluation? 




HEW Institute '68 
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3» What problems , difficulties , etc (procedural , organizational 
analytical) are you encountering and how are you dealing with 




or 

them? 
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. What sorts of research design(s), statistical techniques, and 
data processing systems are you utilizing in your evaluation 
project? What are your available resources for consultation^ 
analysis of your data? To what extent have you planned the 
of your project to include the dimensions of time, resources, 
personnel, processing requirements, etc.? 




and/or 

flow" 
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5. What skills, knowledge, or resource information and materials do 
you now feel you must gain in order to more efficiently and 
effectively pursue your research effort? How do you now see this 
institute as a merits of constructively providing you with these 
necessary skills, knowledge, or resource information and materials 





HEW INSTITUTE IN EDUCATIONAL RESEARCH 

BUREAU OF EDUCATIONAL RESEARCH AND TESTING SERVICES 

P. O. BOX Q 

DURHAM, NEW HAMPSHIRE 03824 



UNIVERSITY OF NEW HAMPSHIRE 
Department of Education 



Telephone: 868-5511. x57 4 
or 668*5322 



Dear Applicant: 

Your interest: in the HEW Institute in Researching and Evaluating Outcomes of 
Educational Innovations in Hew England is gracefully acknowledged, 

A descriptive brochure and application forms are enclosed. Please pay particular 
attention to the instructions listed below: 

1 , The following should be sent , in self -addressed 'envelope provided-, 
to the Director of the Institute. 

a. Application for Admission 

b. Background Data Sheet 

The following should be given to -the applicant's immediate supervisor who 
will complete it and send it to the Director of the Institute. 

a* Confidential Evaluation Form 
b. Self-addressed return envelope 

It is important to note that completed applications must be postmarked no later 
than May 24, 1958. Applications will not be considered complete until all 
other itema- have.been- received. 

We shall notify applicants of the decision of the Selection Committee before 
May 31, 1968. Letters of acceptance from the selected applicants and alternates 
must be postmarked no later than. June 7, 1968. 

Please be certain your completed application forms* and all acu^ompanying items are 
postmarked no later than May 24, 1968. You will be hearing from us during the 
week of May 29, 1968. 

All good wishes for success. 



.Cordially , 

Gilbert R. Austin, Ph.D. 
Institute Director 



GRA/pm 



I DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE 

I OFFICE OF EDUCATION 

| WASHINGTON, D,C. 10:1:2 

f 1 

| CONFIDENTIAL EVALUATION FORM i 

L HEW INSTITUTE IN RESEARCHING AND EVALUATING OUTCOMES OF EDUCATIONAL INNOVATIONS 

I IN NEW ENGLAND 

| ■ (Name of applicant): Name of sponsoring institution: 

t University of New Hampshire 

l ‘ I am seeking admission to the HEW Institute in Researching 
l and Evaluating Outcomes of Educational Inno\ at ions in New England* 



The Selection Committee for the Institute named above has requested that I forward this 
Confidential Evaluation Form to my principal, department chairman, or immediate supervisor, 
Tlease complete the form and return it to the Institute Director* 



1. Name of evaluator: 

Title of position: 

School (or system): 

3 e Considering all the teachers (or specialists) you have worked with or supervised, how 
would you rank the applicant on the following characteristics? 



Above Below CarNt 

Excellent Average Average Average Poor Judge 



a. Ability as a teacher (or specialist) 




. b. Knowledge of subject matter 




c* Effectiveness in working with students 


d„ Effectiveness in working with colleagues 


e* Leadership potential ^ 


f. Scholastic ability; capacity for growth 



4 0 Please provide any comments on the applicants ability, performance, character, 

temperment etc*, which you believe will aid the Selection Committee in determing his 
or her suitability for this Institute „ 



2, How long have you known the applicant 
and in what capacity? 



5. In what ways do you believe that the applicant would benefit from attending this 
Institute? (If the applicant has specific areas of need, please indicate them.) 



6. Does the applicant have a contract, or the offer of a contract, in your school or 
school system for next year? 

Yes No I don’t know (If not, please explain.) 



7. Please comment on ways in which your school oi~ school system may utilize or benefit from 
the training received b}' the applicant if he or she is selected for the Institute,. r;: 



8. Signature of evaluator: 
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